[Doc'd in 2.0.2]1.8.1 & 2.0.2 mixed cluster:Rebalance exited with reason {badarg, [{ns_rebalancer, '-wait_for_memcached/3-lc$^0/1-0-',2}, {ns_rebalancer,wait_for_memcached,

http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-new-rebalance-mixed-cluster/62/consoleFull
./testrunner -i /tmp/rebalance_in.ini get-logs=True,wait_timeout=180,GROUP=P0,EXCLUDE_GROUP=FROM_2_0,get-cbcollect-infrebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... o=True -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,GROUP=IN;P0

mixed suite/cluster:
1.8.1-937-rel
10.3.3.92
10.3.3.94
10.3.3.93

2.0.2-764-rel
10.3.3.99
10.3.3.91
10.3.3.82
10.3.3.97

add to 10.3.3.92(1.8.1) 1*1.8.1(10.3.3.93) and 2*2.0.2(10.3.3.82, 10.3.3.99)

2013-04-14 09:06:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.3.92%2Cns_1%4010.3.3.82%2Cns_1%4010.3.3.99%2Cns_1%4010.3.3.93
2013-04-14 09:06:38 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started
2013-04-14 09:06:38 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0 %
2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
2013-04-14 09:06:48 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] Latest logs from UI:
2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.99', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. Will try to take over.", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956005954.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.82', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. Will try to take over.", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956005954.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.93', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.93\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1365956004357.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] {u'node': u'ns_1@10.3.3.92', u'code': 2, u'text': u"Rebalance exited with reason {badarg,\n [

{ns_rebalancer,\n '-wait_for_memcached/3-lc$^0/1-0-',2}

,\n

{ns_rebalancer,wait_for_memcached,3}

,\n

{ns_rebalancer,'-rebalance/3-fun-0-',5}

,\n

{lists,foreach,2}

,\n

{ns_rebalancer,rebalance,3}

]}\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1365956004324.0, u'type': u'info'}
2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.99', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. But I won't try to take over since rebalance seems to be running", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956003954.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.82', u'code': 0, u'text': u"Candidate got master heartbeat from node 'ns_1@10.3.3.92' which has lower priority. But I won't try to take over since rebalance seems to be running", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1365956003954.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.82', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.82\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1365956003367.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.99', u'code': 1, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.3.99\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1365956003359.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.92', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1365956003298.0, u'type': u'info'}

2013-04-14 09:06:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_progress]

{u'node': u'ns_1@10.3.3.92', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.3.92','ns_1@10.3.3.82',\n 'ns_1@10.3.3.99','ns_1@10.3.3.93'], EjectNodes = []\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1365956003257.0, u'type': u'info'}