[Windows] Rebalance exited with reason : buckets_shutdown_wait_failed (after failover)

[2015-06-30 14:38:11,884] - [xdcrnewbasetests:1829] INFO - Starting failover for nodes:[ip:172.23.107.68 port:8091 ssh_username:Administrator] at C1 cluster 172.23.107.67

[2015-06-30 14:38:12,701] - [task:2990] INFO - Failing over 172.23.107.68:8091 with graceful=False

[2015-06-30 14:38:13,967] - [rest_client:1111] INFO - fail_over node ns_1@172.23.107.68 successful

[2015-06-30 14:38:13,967] - [task:2970] INFO - 0 seconds sleep after failover, for nodes to go pending....

[2015-06-30 14:38:14,000] - [rest_client:1144] INFO - add_back_node ns_1@172.23.107.68 successful

[2015-06-30 14:38:15,011] - [rest_client:1166] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.67%2Cns_1%40172.23.107.68

[2015-06-30 14:38:15,023] - [rest_client:1170] INFO - rebalance operation started

[2015-06-30 14:38:15,033] - [rest_client:1288] INFO - rebalance percentage : 0.00 %

[2015-06-30 14:38:25,052] - [rest_client:1288] INFO - rebalance percentage : 0.00 %

[2015-06-30 14:38:35,077] - [rest_client:1271] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed

[2015-06-30 14:38:36,017] - [rest_client:2195] INFO - Latest logs from UI on 172.23.107.67:

[2015-06-30 14:38:36,017] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.67', u'code': 2, u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n                              [{\'ns_1@172.23.107.68\',\n                                {\'EXIT\',\n                                 {old_buckets_shutdown_wait_failed,\n                                  ["standard_bucket_1"]}}}]}\n', u'shortText': u'message', u'serverTime': u'2015-06-30T14:38:13.418Z', u'module': u'ns_orchestrator', u'tstamp': 1435700293418, u'type': u'info'}

[2015-06-30 14:38:36,018] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.67', u'code': 0, u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.107.68\',\n                                                         {\'EXIT\',\n                                                          {old_buckets_shutdown_wait_failed,\n                                                           ["standard_bucket_1"]}}}]\n', u'shortText': u'message', u'serverTime': u'2015-06-30T14:38:13.418Z', u'module': u'ns_rebalancer', u'tstamp': 1435700293418, u'type': u'critical'}

[2015-06-30 14:38:36,018] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.68', u'code': 0, u'text': u'Shutting down bucket "standard_bucket_1" on \'ns_1@172.23.107.68\' for deletion', u'shortText': u'message', u'serverTime': u'2015-06-30T14:38:07.492Z', u'module': u'ns_memcached', u'tstamp': 1435700287492, u'type': u'info'}

[2015-06-30 14:38:36,018] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.68', u'code': 0, u'text': u'Shutting down bucket "standard_bucket_2" on \'ns_1@172.23.107.68\' for deletion', u'shortText': u'message', u'serverTime': u'2015-06-30T14:37:57.196Z', u'module': u'ns_memcached', u'tstamp': 1435700277196, u'type': u'info'}

[2015-06-30 14:38:36,018] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.67', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.67','ns_1@172.23.107.68'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2015-06-30T14:37:53.388Z', u'module': u'ns_orchestrator', u'tstamp': 1435700273388, u'type': u'info'}

[2015-06-30 14:38:36,019] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.68', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.107.68\' for deletion', u'shortText': u'message', u'serverTime': u'2015-06-30T14:37:52.672Z', u'module': u'ns_memcached', u'tstamp': 1435700272672, u'type': u'info'}

[2015-06-30 14:38:36,019] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.67', u'code': 0, u'text': u"Failed over 'ns_1@172.23.107.68': ok", u'shortText': u'message', u'serverTime': u'2015-06-30T14:37:52.327Z', u'module': u'ns_rebalancer', u'tstamp': 1435700272327, u'type': u'info'}

[2015-06-30 14:38:36,019] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.67', u'code': 0, u'text': u"Starting failing over 'ns_1@172.23.107.68'", u'shortText': u'message', u'serverTime': u'2015-06-30T14:37:51.063Z', u'module': u'ns_rebalancer', u'tstamp': 1435700271063, u'type': u'info'}

[2015-06-30 14:38:36,019] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.68', u'code': 0, u'text': u"Service 'goxdcr' exited with status 1. Restarting. Messages: net/http.(*persistConn).readLoop(0xc08414ca50)\n\tc:/go/src/net/http/transport.go:842 +0xab\ncreated by net/http.(*Transport).dialConn\n\tc:/go/src/net/http/transport.go:660 +0xca6\n[goport] 2015/06/30 14:37:05 c:/Program Files/Couchbase/Server/bin/goxdcr.exe terminated: exit status 2", u'shortText': u'message', u'serverTime': u'2015-06-30T14:37:05.076Z', u'module': u'ns_log', u'tstamp': 1435700225076, u'type': u'info'}

[2015-06-30 14:38:36,020] - [rest_client:2196] ERROR - {u'node': u'ns_1@172.23.107.67', u'code': 0, u'text': u'Replication from bucket "standard_bucket_2" to bucket "standard_bucket_2" on cluster "remote_cluster_C1-C2" created.', u'shortText': u'message', u'serverTime': u'2015-06-30T14:36:52.734Z', u'module': u'xdcr', u'tstamp': 1435700212734, u'type': u'info'}