Description
Build
5.1.0-1368
Please note that the same test passed on a previous run on the same build. The rebalance failure is most probably caused by a race condition. This may not be reproducible with the following testcase.
Rebalance fails during the teardown phase when we remove .151 from .122 cluster. Please note, all subsequent attempts to rebalance also fail, leaving the cluster in a bad state.
Test
./testrunner -i /tmp/testexec.3801.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=1000,cluster=D,F,standard_buckets=3,sasl_buckets=3,index_per_bucket=3,update=True,expires=30,memory_only=True,GROUP=P0
[2017-11-08 12:28:00,073] - [rest_client:1548] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed |
[2017-11-08 12:28:00,084] - [rest_client:2983] INFO - Latest logs from UI on 172.23.105.122: |
[2017-11-08 12:28:00,084] - [rest_client:2984] ERROR - {u'node': u'ns_1@172.23.105.122', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n {rebalance_failed,\n {service_error,\n <<"planner: CfgGetPlanPIndexes err: cfg_metakv_lean: getLeanPlan, hash mismatch between plan contents: 784da5db75932c7927927ffc66bdadc0, and directory stamp: 1665e589df23639e33a9317463c8f2e6\\nplanner: CfgGetPlanPIndexes err: cfg_metakv_lean: getLeanPlan, hash mismatch between plan contents: 784da5db75932c7927927ffc66bdadc0, and directory stamp: 1665e589df23639e33a9317463c8f2e6">>}}}', u'shortText': u'message', u'serverTime': u'2017-11-08T12:27:50.118Z', u'module': u'ns_orchestrator', u'tstamp': 1510172870118, u'type': u'critical'} |
[2017-11-08 12:28:00,085] - [rest_client:2984] ERROR - {u'node': u'ns_1@172.23.105.122', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.122'], EjectNodes = ['ns_1@172.23.105.151'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2017-11-08T12:27:50.052Z', u'module': u'ns_orchestrator', u'tstamp': 1510172870052, u'type': u'info'} |
Attaching logs.
Attachments
Issue Links
- relates to
-
MB-44485 [System Test][ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan and directory
-
- Closed
-