Rebalance exited with reason {buckets_shutdown_wait_failed} during bucket deletion in warmup Test

Failure reported towards the end of warmup Test:

 deleting existing buckets [] on 172.23.106.250

2016-06-27 15:36:08 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes

2016-06-27 15:36:08 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.249%2Cns_1%40172.23.106.247%2Cns_1%40172.23.106.250&user=Administrator&knownNodes=ns_1%40172.23.106.249%2Cns_1%40172.23.106.247%2Cns_1%40172.23.106.248%2Cns_1%40172.23.106.250

2016-06-27 15:36:08 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started

2016-06-27 15:36:08 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %

2016-06-27 15:36:18 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 %

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client._rebalance_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed

2016-06-27 15:36:28 | INFO | MainProcess | test_thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.106.248:

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.247', u'code': 2, u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n                              [{\'ns_1@172.23.106.248\',\n                                {\'EXIT\',\n                                 {old_buckets_shutdown_wait_failed,\n                                  ["default"]}}}]}\n', u'shortText': u'message', u'serverTime': u'2016-06-27T15:36:28.135Z', u'module': u'ns_orchestrator', u'tstamp': 1467066988135, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.247', u'code': 0, u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.106.248\',\n                                                         {\'EXIT\',\n                                                          {old_buckets_shutdown_wait_failed,\n                                                           ["default"]}}}]\n', u'shortText': u'message', u'serverTime': u'2016-06-27T15:36:28.135Z', u'module': u'ns_rebalancer', u'tstamp': 1467066988135, u'type': u'critical'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.247', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.106.248'], EjectNodes = ['ns_1@172.23.106.249',\n                                                                       'ns_1@172.23.106.247',\n                                                                       'ns_1@172.23.106.250'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2016-06-27T15:36:08.120Z', u'module': u'ns_orchestrator', u'tstamp': 1467066968120, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.248', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.106.248\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.106.249\', u\'ns_1@172.23.106.247\', u\'ns_1@172.23.106.250\']\n', u'shortText': u'client-side error report', u'serverTime': u'2016-06-27T15:36:08.068Z', u'module': u'menelaus_web', u'tstamp': 1467066968068, u'type': u'warning'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.250', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.106.250\' for deletion', u'shortText': u'message', u'serverTime': u'2016-06-27T15:35:38.276Z', u'module': u'ns_memcached', u'tstamp': 1467066938276, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.247', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.106.247\' for deletion', u'shortText': u'message', u'serverTime': u'2016-06-27T15:35:38.179Z', u'module': u'ns_memcached', u'tstamp': 1467066938179, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.248', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.106.248\' for deletion', u'shortText': u'message', u'serverTime': u'2016-06-27T15:35:38.063Z', u'module': u'ns_memcached', u'tstamp': 1467066938063, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.249', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.106.249\' for deletion', u'shortText': u'message', u'serverTime': u'2016-06-27T15:35:37.937Z', u'module': u'ns_memcached', u'tstamp': 1467066937937, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.250', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.250\' in 1 seconds.', u'shortText': u'message', u'serverTime': u'2016-06-27T15:11:45.457Z', u'module': u'ns_memcached', u'tstamp': 1467065505457, u'type': u'info'}

2016-06-27 15:36:28 | ERROR | MainProcess | test_thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.106.249', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.106.249\' in 1 seconds.', u'shortText': u'message', u'serverTime': u'2016-06-27T15:11:37.240Z', u'module': u'ns_memcached', u'tstamp': 1467065497240, u'type': u'info'}

Traceback (most recent call last):

  File "pytests/basetestcase.py", line 150, in setUp

    self._cluster_cleanup()

  File "pytests/basetestcase.py", line 379, in _cluster_cleanup

    ClusterOperationHelper.cleanup_cluster(self.servers, master=self.master)

  File "lib/membase/helper/cluster_helper.py", line 260, in cleanup_cluster

    wait_for_rebalance=wait_for_rebalance)

  File "lib/membase/api/rest_client.py", line 112, in remove_nodes

    return self.rest.monitorRebalance()

  File "lib/membase/api/rest_client.py", line 1343, in monitorRebalance

    progress = self._rebalance_progress()

  File "lib/membase/api/rest_client.py", line 1403, in _rebalance_progress

    raise RebalanceFailedException(msg)

RebalanceFailedException: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed

Cluster instance shutdown with force

ERROR