Bucket deletion timeout issue followed by rebalance failures found in XDCR tests

[2017-03-05 11:23:17,744] - [cluster_helper:78] INFO - waiting for ns_server @ 172.23.105.181:8091

[2017-03-05 11:23:17,750] - [cluster_helper:80] INFO - ns_server @ 172.23.105.181:8091 is running

Cluster instance shutdown with force

[2017-03-05 11:23:17,750] - [xdcrnewbasetests:1149] INFO - removing xdcr/nodes settings

[2017-03-05 11:23:17,813] - [rest_client:1025] INFO - removing remote cluster name:remote_cluster_C2-C1

[2017-03-05 11:23:17,848] - [xdcrnewbasetests:1159] INFO - cleanup [ip:172.23.107.120 port:8091 ssh_username:root, ip:172.23.107.169 port:8091 ssh_username:root, ip:172.23.105.180 port:8091 ssh_username:root]

[2017-03-05 11:23:17,886] - [bucket_helper:138] INFO - deleting existing buckets [u'default'] on 172.23.107.120

[2017-03-05 11:23:17,886] - [bucket_helper:140] INFO - remove bucket default ...

[2017-03-05 11:23:47,906] - [rest_client:790] ERROR - http://172.23.107.120:8091/pools/default/buckets/default error 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"}

[2017-03-05 11:23:47,906] - [rest_client:1898] WARNING - Bucket deletion timed out waiting for all nodes

[2017-03-05 11:23:47,931] - [bucket_helper:153] ERROR - Unable to get timings for bucket

[2017-03-05 11:23:47,931] - [bucket_helper:154] INFO - deleted bucket : default from 172.23.107.120

[2017-03-05 11:23:47,931] - [bucket_helper:230] INFO - waiting for bucket deletion to complete....

[2017-03-05 11:23:47,955] - [rest_client:134] INFO - node 172.23.107.120 existing buckets : []

[2017-03-05 11:23:48,005] - [cluster_helper:255] INFO - rebalancing all nodes in order to remove nodes

[2017-03-05 11:23:48,010] - [rest_client:1322] INFO - rebalance params : password=welcome&ejectedNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180&user=Administrator&knownNodes=ns_1%40172.23.107.169%2Cns_1%40172.23.105.180%2Cns_1%40172.23.107.120

[2017-03-05 11:23:48,017] - [rest_client:1326] INFO - rebalance operation started

[2017-03-05 11:23:48,047] - [rest_client:1457] INFO - rebalance percentage : 0.00 %

[2017-03-05 11:23:58,064] - [rest_client:1457] INFO - rebalance percentage : 0.00 %

[2017-03-05 11:24:08,083] - [rest_client:1440] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed

[2017-03-05 11:24:08,208] - [rest_client:2726] INFO - Latest logs from UI on 172.23.107.120:

[2017-03-05 11:24:08,208] - [rest_client:2727] ERROR - {u'node': u'ns_1@172.23.107.120', u'code': 0