Details
-
Task
-
Resolution: Fixed
-
Major
-
2.1.0
-
Security Level: Public
-
None
-
CentOS release 5.8, 2.0.2-764-re
Description
http://qa.hq.northscale.net/view/2.0.1/job/ubuntu-64-2.0-new-rebalance-tests-P0/415/console
./testrunner -i /tmp/rebalance-tests.ini wait_timeout=180,GROUP=P0,get-cbcollect-info=True,stop-on-failure=True -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ddoc_compaction,items=500000,max_verify=100000,fragmentation_value=60,GROUP=OUT;BASIC;COMPACTION;P0;FROM_2_0
test with ddoc compaction:
default bucket with 500000 items, 1ddoc*5views
[2013-04-16 05:50:02,412] - [task:1753] INFO - design doc ddoc1 is compacting
[2013-04-16 05:50:07,490] - [task:1748] INFO - stats compaction: (1,45.2233728939)
[2013-04-16 05:50:07,491] - [task:1756] INFO - compactor was run, compaction revision was changed on 1
[2013-04-16 05:50:07,491] - [task:1759] INFO - fragmentation went from 69 to 45
[2013-04-16 05:50:09,511] - [rest_client:940] INFO - rebalance percentage : 15.9197945499 %
[2013-04-16 05:50:19,525] - [rest_client:940] INFO - rebalance percentage : 33.5959873768 %
[2013-04-16 05:50:29,534] - [rest_client:940] INFO - rebalance percentage : 52.8221587537 %
[2013-04-16 05:50:39,542] - [rest_client:940] INFO - rebalance percentage : 66.4929596916 %
[2013-04-16 05:50:50,558] - [rest_client:940] INFO - rebalance percentage : 75.6601053108 %
[2013-04-16 05:51:01,563] - [rest_client:923] ERROR -
- rebalance failed
[2013-04-16 05:51:01,563] - [rest_client:924] INFO - Latest logs from UI:
[2013-04-16 05:51:01,659] - [rest_client:925] ERROR - {u'node': u'ns_1@10.5.2.13', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n {\'EXIT\',<0.9415.34>,\n {{{{badmatch,\n {error,\n
}},\n [
{capi_set_view_manager,handle_call,3},\n {gen_server,handle_msg,5},\n {gen_server,init_it,6},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [\'capi_set_view_manager-default\',\n {wait_index_updated,375},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\'ns_1@10.5.2.13\'},\n {if_rebalance,<0.31446.33>,\n {wait_index_updated,375}},\n infinity]}}}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1366117462189, u'type': u'info'}[2013-04-16 05:51:01,659] - [rest_client:925] ERROR - {u'node': u'ns_1@10.5.2.13', u'code': 0, u'text': u'<0.9278.34> exited with {unexpected_exit,\n {\'EXIT\',<0.9415.34>,\n {{badmatch,\n {error,\n {error,\n <<"Partition 375 not in active nor passive set">>},\n [{capi_set_view_manager,handle_call,3}
,\n
{gen_server,handle_msg,5},\n
{gen_server,init_it,6},\n
{proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [\'capi_set_view_manager-default\',\n
{wait_index_updated,375},\n infinity]}},\n {gen_server,call,\n [
{\'janitor_agent-default\',\'ns_1@10.5.2.13\'},\n {if_rebalance,<0.31446.33>,\n {wait_index_updated,375}},\n infinity]}}}}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1366117462002, u'type': u'critical'}
[2013-04-16 05:51:01,660] - [rest_client:925] ERROR -
[2013-04-16 05:51:01,660] - [rest_client:925] ERROR - {u'node': u'ns_1@10.3.121.66', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` failed:
{index_compactor_died_too_soon,\n <<"default">>,\n <<"_design/ddoc1">>,\n main}. See logs for detailed reason.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117427129, u'type': u'critical'}
[2013-04-16 05:51:01,660] - [rest_client:925] ERROR -
[2013-04-16 05:51:01,661] - [rest_client:925] ERROR -
{u'node': u'ns_1@10.5.2.15', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` completed.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117421694, u'type': u'info'}[2013-04-16 05:51:01,661] - [rest_client:925] ERROR -
{u'node': u'ns_1@10.3.121.63', u'code': 0, u'text': u'User-triggered compaction of view `default/_design/ddoc1` completed.', u'shortText': u'message', u'module': u'compaction_daemon', u'tstamp': 1366117408988, u'type': u'info'}[2013-04-16 05:51:01,661] - [rest_client:925] ERROR -
{u'node': u'ns_1@10.5.2.13', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1366117402503, u'type': u'info'}[2013-04-16 05:51:01,661] - [rest_client:925] ERROR -
{u'node': u'ns_1@10.5.2.13', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1366117402122, u'type': u'info'}[2013-04-16 05:51:01,661] - [rest_client:925] ERROR -
{u'node': u'ns_1@10.5.2.13', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.5.2.14','ns_1@10.5.2.15',\n 'ns_1@10.3.121.63','ns_1@10.3.121.69',\n 'ns_1@10.5.2.13','ns_1@10.3.121.64'], EjectNodes = ['ns_1@10.3.121.66']\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1366117402023, u'type': u'info'}Attachments
For Gerrit Dashboard: MB-8109 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
25725,2 | MB-8109 Set pending transition for outdated updater groups | 2.0.2 | couchdb | Status: MERGED | +2 | +1 |
25727,1 | Merge remote-tracking branch 'origin/2.0.2' | master | couchdb | Status: MERGED | +2 | +1 |