[Windwos]Rebalance exited with reason {{bulk_set_vbucket_state_failed, [{'ns_1@IP',{'EXIT', {{{{unexpected_reason,{{badmatch,{error,closed}},

http://qa.hq.northscale.net/job/windows_rebalance_regression_P1/63/consoleFull

./testrunner -i /tmp/windows_rebalance_regression_P1.ini get-cbcollect-info=True,EXCLUDE_GROUP=NOT_WIND -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=3,num-swap=2,keys-count=1000000

[2013-07-02 02:52:38,552] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.97.59%2Cns_1%4010.3.3.176%2Cns_1%4010.3.2.242%2Cns_1%40172.23.97.60%2Cns_1%40172.23.97.58%2Cns_1%4010.3.2.131%2Cns_1%4010.3.2.241
....

[2013-07-02 03:11:01,720] - [rest_client:1031] INFO - rebalance percentage : 22.9714110169 %
[2013-07-02 03:11:04,738] - [rest_client:1014] ERROR -

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
[2013-07-02 03:11:04,738] - [rest_client:1015] INFO - Latest logs from UI:
[2013-07-02 03:11:05,309] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.2.131', u'code': 2, u'text': u"Rebalance exited with reason bulk_set_vbucket_state_failed,\n [{'ns_1@172.23.97.60',\n {'EXIT',\n {{{{unexpected_reason,\n {{badmatch,{error,closed,\n [

{mc_binary,quick_stats_recv,3}

,\n

{mc_binary,quick_stats_loop,5}

,\n

{mc_binary,quick_stats,5}

,\n

{mc_client_binary,\n get_zero_open_checkpoint_vbuckets,3}

,\n

{ebucketmigrator_srv,handle_call,3}

,\n

{gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]}},\n [{misc,executing_on_new_process,1},\n {tap_replication_manager,\n change_vbucket_filter,4},\n {tap_replication_manager,\n '-do_set_incoming_replication_map/3-lc$^2/1-2-',\n 2},\n {tap_replication_manager,\n do_set_incoming_replication_map,3},\n {tap_replication_manager,handle_call,3},\n {gen_server,handle_msg,5}

,\n

{proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n ['tap_replication_manager-bucket-2',\n {change_vbucket_replication,633,\n 'ns_1@10.3.2.241'},\n infinity]}},\n {gen_server,call,\n [{'janitor_agent-bucket-2',\n 'ns_1@172.23.97.60'},\n {if_rebalance,<0.14393.4>,\n {update_vbucket_state,633,replica,\n undefined,'ns_1@10.3.2.241'}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4},\n {ns_vbucket_mover,\n update_replication_post_move,3},\n {ns_vbucket_mover,on_move_done,2},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}

]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1372760889817, u'type': u'info'}
[2013-07-02 03:11:05,310] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'Bucket "bucket-2" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1372759789442, u'type': u'info'}

[2013-07-02 03:11:05,310] - [rest_client:1016] ERROR -

{u'node': u'ns_1@172.23.97.58', u'code': 1, u'text': u'Bucket "bucket-2" loaded on node \'ns_1@172.23.97.58\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1372759785609, u'type': u'info'}

[2013-07-02 03:11:05,310] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.2.131', u'code': 0, u'text': u'Started rebalancing bucket bucket-2', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1372759785567, u'type': u'info'}

[2013-07-02 03:11:05,311] - [rest_client:1016] ERROR -

{u'node': u'ns_1@172.23.97.60', u'code': 1, u'text': u'Bucket "bucket-2" loaded on node \'ns_1@172.23.97.60\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1372759785097, u'type': u'info'}

[2013-07-02 03:11:05,311] - [rest_client:1016] ERROR -

{u'node': u'ns_1@172.23.97.59', u'code': 1, u'text': u'Bucket "bucket-2" loaded on node \'ns_1@172.23.97.59\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1372759785067, u'type': u'info'}

[2013-07-02 03:11:05,311] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.2.131', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.97.59','ns_1@10.3.3.176',\n 'ns_1@10.3.2.242','ns_1@172.23.97.60',\n 'ns_1@172.23.97.58','ns_1@10.3.2.131',\n 'ns_1@10.3.2.241'], EjectNodes = []\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1372759784880, u'type': u'info'}

[2013-07-02 03:11:05,312] - [rest_client:1016] ERROR -

{u'node': u'ns_1@172.23.97.58', u'code': 0, u'text': u'Deleting old data files of bucket "default"', u'shortText': u'message', u'module': u'ns_storage_conf', u'tstamp': 1372759784579, u'type': u'info'}

[2013-07-02 03:11:05,312] - [rest_client:1016] ERROR -

{u'node': u'ns_1@172.23.97.59', u'code': 0, u'text': u'Deleting old data files of bucket "default"', u'shortText': u'message', u'module': u'ns_storage_conf', u'tstamp': 1372759784553, u'type': u'info'}

[2013-07-02 03:11:05,313] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.2.131', u'code': 4, u'text': u"Node 'ns_1@10.3.2.131' saw that node 'ns_1@10.3.2.241' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1372759783677, u'type': u'info'}

[2013-07-02 03:11:05,314] - [rebalance_helper:481] ERROR - rebalance failed: Rebalance Failed:

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed