Details
-
Bug
-
Resolution: Fixed
-
Critical
-
3.0
-
Security Level: Public
-
None
-
1:10.3.5.115
2:10.3.5.116
3:10.3.5.117
4:10.3.5.118
5:10.6.2.185
6:10.6.2.186
7:10.5.3.5
-
Untriaged
-
-
Yes
-
June 30 - July 18
Description
3.0.0-1139, Centos 6x
Test Case:: ./testrunner i centos_x64-rebalance_out.ini get-cbcollect-info=False,get-logs=False,stop-on-failure=False,get-coredumps=False,force_kill_memached=False,verify_unacked_bytes=True,total_vbuckets=128,std_vbucker_dist=8,chk_max_items=1000,chk_period=30 -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_with_ops,nodes_out=3,replicas=3,items=100000,doc_ops=create:update:delete,max_verify=100000,value_size=1024,GROUP=OUT;P0
1. Create 7 node cluster
2. Create default bucket and add 100K items
3. Rebalance-out 3 Nodes with mutations running in parallel
Rebalance in Step 3 Exited with the following exception
u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2014-08-12 19:25:32,038] - [rest_client:2010] INFO - Latest logs from UI on 10.3.5.115:
[2014-08-12 19:25:32,039] - [rest_client:2011] ERROR -
[2014-08-12 19:25:32,040] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.115', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.31764.3>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.5.117',\n {'EXIT',\n badmatch,{error,closed,\n {gen_server,call,\n [
{'janitor_agent-default',\n 'ns_1@10.3.5.117'},\n {if_rebalance,<0.28660.3>,\n {update_vbucket_state,79,replica,\n passive,'ns_1@10.5.3.5'}},\n infinity]}}}}]}}}\n", u'shortText': u'message', u'serverTime': u'2014-08-12T19:26:13.725Z', u'module': u'ns_orchestrator', u'tstamp': 1407896773725, u'type': u'info'}
[2014-08-12 19:25:32,040] - [rest_client:2011] ERROR - {u'node': u'ns_1@10.3.5.115', u'code': 0, u'text': u"<0.31751.3> exited with {unexpected_exit,\n {'EXIT',<0.31764.3>,\n {bulk_set_vbucket_state_failed,\n [{'ns_1@10.3.5.117',\n {'EXIT',\n badmatch,{error,closed,\n {gen_server,call,\n [
,\n {if_rebalance,<0.28660.3>,\n {update_vbucket_state,79,replica,passive,\n 'ns_1@10.5.3.5'}},\n infinity]}}}}]}}}", u'shortText': u'message', u'serverTime': u'2014-08-12T19:26:13.718Z', u'module': u'ns_vbucket_mover', u'tstamp': 1407896773718, u'type': u'critical’}