Details
-
Bug
-
Resolution: Fixed
-
Major
-
5.0.0
-
Untriaged
-
No
Description
Build
5.0.0-2068
Testcase
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P0 -t fts.moving_topology_fts.MovingTopFTS.swap_rebalance_during_index_building,items=1000,cluster=D,F,F,replicas=0,GROUP=P0
Swap-rebalance [remove_node:[ip:172.23.106.66 port:8091] -> [add_node:[ip:172.23.105.54 port:8091 ssh_username:root]] at cluster 172.23.105.96 results in rebalance failure.
Swap rebalance of fts node, during rebalance failed with
| [2017-02-21 10:37:26,267] - [rest_client:2709] ERROR - {u'node': u'ns_1@172.23.105.96', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n {lost_connection,shutdown}}', u'shortText': u'message', u'serverTime': u'2017-02-21T10:37:17.019Z', u'module': u'ns_orchestrator', u'tstamp': 1487702237019, u'type': u'critical'{color:#000000}} |
| [2017-02-21 10:37:26,267] - [rest_client:2709] ERROR - {u'node': u'ns_1@172.23.105.96', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2017-02-21T10:37:16.385Z', u'module': u'ns_vbucket_mover', u'tstamp': 1487702236385, u'type': u'info'{color:#000000}} |
| [2017-02-21 10:37:26,268] - [rest_client:2709] ERROR - {u'node': u'ns_1@172.23.105.96', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2017-02-21T10:37:16.228Z', u'module': u'ns_rebalancer', u'tstamp': 1487702236228, u'type': u'info'{color:#000000}} |
| [2017-02-21 10:37:26,268] - [rest_client:2709] ERROR - {u'node': u'ns_1@172.23.105.96', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.54','ns_1@172.23.105.96',\n 'ns_1@172.23.105.190'], EjectNodes = ['ns_1@172.23.106.66'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2017-02-21T10:37:16.146Z', u'module': u'ns_orchestrator', u'tstamp': 1487702236146, u'type': u'info'{color:#000000}} |
Multiple rebalances across fts jobs are now failing. Steve had warned that the recent changes that went into build 2018 will have some effect on rebalance and failover.
Adding logs to this ticket.
Please feel free to assign to Steve if this failure relates to his changes.
