Rebalance in operation exited with reason {bulk_set_vbucket_state_failed}

http://qa.hq.northscale.net/job/centos_x64--107_01--rebalanceXDCR-P1/10/consoleFull

[Test]
./testrunner i centos_x64107_01-rebalanceXDCR-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=bidirection,ctopology=chain,doc-ops=update-delete,doc-ops-dest=update-delete,expires=60,rebalance=source-destination,num_rebalance=1,GROUP=P1

[Test Error]
[2014-07-02 03:59:21,417] - [rest_client:1208] INFO - rebalance percentage : 83.2327519408 %
[2014-07-02 03:59:22,427] - [rest_client:1192] ERROR -

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
[2014-07-02 03:59:22,470] - [rest_client:1999] INFO - Latest logs from UI on 10.5.2.231:
[2014-07-02 03:59:22,470] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.5.2.231', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n {\'EXIT\',<0.15215.28>,\n {bulk_set_vbucket_state_failed,\n [{\'ns_1@10.3.5.68\',\n {\'EXIT\',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7,\n [

{file,"src/janitor_agent.erl"},\n {line,394}]},\n {janitor_agent,\n \'bulk_set_vbucket_state/4-fun-0\',4,\n [{file,"src/janitor_agent.erl"}

,\n

{line,381}]},\n {misc,\'parallel_map/3-fun-0\',4,\n [{file,"src/misc.erl"},\n {line,86}]}]}}}]}}}\n', u'shortText': u'message', u'serverTime': u'2014-07-02T03:59:20.335Z', u'module': u'ns_orchestrator', u'tstamp': 1404298760335, u'type': u'info'}
[2014-07-02 03:59:22,472] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.5.2.231', u'code': 0, u'text': u'<0.6273.28> exited with {unexpected_exit,\n {\'EXIT\',<0.15215.28>,\n {bulk_set_vbucket_state_failed,\n [{\'ns_1@10.3.5.68\',\n {\'EXIT\',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7,\n [{file,"src/janitor_agent.erl"},{line,394}]},\n {janitor_agent,\n \'bulk_set_vbucket_state/4-fun-0\',4,\n [{file,"src/janitor_agent.erl"},{line,381}

]},\n {misc,\'parallel_map/3-fun-0\',4,\n [

{file,"src/misc.erl"}

,

{line,86}

]}]}}}]}}}', u'shortText': u'message', u'serverTime': u'2014-07-02T03:59:20.328Z', u'module': u'ns_vbucket_mover', u'tstamp': 1404298760328, u'type': u'critical'}
[2014-07-02 03:59:22,473] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.5.68', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.68\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-07-02T03:59:19.689Z', u'module': u'ns_memcached', u'tstamp': 1404298759689, u'type': u'info'}

[2014-07-02 03:59:22,474] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.5.68', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@10.3.5.68\'. Version: "3.0.0-918-rel-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2014-07-02T03:59:19.612Z', u'module': u'menelaus_sup', u'tstamp': 1404298759612, u'type': u'info'}

[2014-07-02 03:59:22,475] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.5.68', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@10.3.5.68\' for server shutdown', u'shortText': u'message', u'serverTime': u'2014-07-02T03:59:19.443Z', u'module': u'ns_memcached', u'tstamp': 1404298759443, u'type': u'info'}

[2014-07-02 03:59:22,476] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.5.2.231', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:23.446Z', u'module': u'ns_vbucket_mover', u'tstamp': 1404298583446, u'type': u'info'}

[2014-07-02 03:59:22,477] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.5.68', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.68\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:22.539Z', u'module': u'ns_memcached', u'tstamp': 1404298582539, u'type': u'info'}

[2014-07-02 03:59:22,478] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.5.2.231', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:21.089Z', u'module': u'ns_rebalancer', u'tstamp': 1404298581089, u'type': u'info'}

[2014-07-02 03:59:22,479] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.5.2.231', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.5.2.233','ns_1@10.5.2.231',\n 'ns_1@10.5.2.232','ns_1@10.3.5.68'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:20.857Z', u'module': u'ns_orchestrator', u'tstamp': 1404298580857, u'type': u'info'}

[2014-07-02 03:59:22,479] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.5.68', u'code': 3, u'text': u'Node ns_1@10.3.5.68 joined cluster', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:20.807Z', u'module': u'ns_cluster', u'tstamp': 1404298580807, u'type': u'info'}

[Operation failed]
adding remote node @10.3.5.68:8091 to this cluster @10.5.2.231:8091