[CentOS] Online upgrade failed 2.2.0-837 -> 3.0.0-918 due to rebalance failed with badarg,\n [{erlang,\'++\',\n [{\'EXIT\',\n {{function_clause,\n [{janitor_agent,handle_call,\n [{get_mass_upr_docs_estimate,[]}

http://qa.hq.northscale.net/job/centos_x64--104_01--XDCR_upgrade-P1/16/consoleFull

[Test Error]
[2014-07-03 17:31:30,321] - [rest_client:1399] INFO - Node versions in cluster [u'2.2.0-837-rel-enterprise', u'2.2.0-837-rel-enterprise']
[2014-07-03 17:31:30,339] - [rest_client:1399] INFO - Node versions in cluster [u'3.0.0-918-rel-enterprise']
[2014-07-03 17:31:31,293] - [task:284] INFO - adding node 10.3.121.199:8091 to cluster
[2014-07-03 17:31:31,293] - [rest_client:933] INFO - adding remote node @10.3.121.199:8091 to this cluster @10.3.3.126:8091
[2014-07-03 17:31:31,661] - [rest_client:1087] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.121.199%2Cns_1%4010.3.3.126%2Cns_1%4010.3.5.11
[2014-07-03 17:31:41,726] - [rest_client:1091] INFO - rebalance operation started
[2014-07-03 17:31:41,731] - [rest_client:1208] INFO - rebalance percentage : 0 %
[2014-07-03 17:31:51,740] - [rest_client:1192] ERROR -

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
[2014-07-03 17:31:51,847] - [rest_client:1999] INFO - Latest logs from UI on 10.3.3.126:
[2014-07-03 17:31:51,847] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.3.121.199', u'code': 2, u'text': u'Rebalance exited with reason {badarg,\n [{erlang,\'++\',\n [{\'EXIT\',\n {{function_clause,\n [{janitor_agent,handle_call,\n [

{get_mass_upr_docs_estimate,[]},\n {<0.1293.0>,#Ref<0.0.0.13844>},\n {state,"default",<0.1299.0>,\n #Ref<17408.0.1.261106>,[],\n [active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica],\n [undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n passive,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined],\n 0}]},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@10.3.3.126\'},\n {get_mass_upr_docs_estimate,[]}

,\n infinity]}}},\n {\'EXIT\',\n {{function_clause,\n [{janitor_agent,handle_call,\n [

{get_mass_upr_docs_estimate,[]},\n {<0.1294.0>,#Ref<0.0.0.13747>},\n {state,"default",<0.1299.0>,\n #Ref<17413.0.1.257063>,[],\n [replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active],\n [undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined],\n 0}]},\n {gen_server,handle_msg,5},\n {proc_lib,init_p_do_apply,3}]},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@10.3.5.11\'},\n {get_mass_upr_docs_estimate,[]}

,\n infinity]}}}],\n []},\n {lists,append,1,\n [

{file,"lists.erl"},{line,125}]},\n {lists,append,1,\n [{file,"lists.erl"}

,

{line,125}

]},\n {ns_rebalance_observer,\n initiate_bucket_rebalance,2,\n [

{file,"src/ns_rebalance_observer.erl"},\n {line,209}]},\n {ns_rebalance_observer,handle_set_ff_map,2,\n [{file,"src/ns_rebalance_observer.erl"}

,\n

{line,257}

]},\n {ns_rebalance_observer,handle_cast,2,\n [

{file,"src/ns_rebalance_observer.erl"}

,\n

{line,105}

]},\n {gen_server,handle_msg,5,\n [

{file,"gen_server.erl"}

,

{line,604}

]},\n {proc_lib,init_p_do_apply,3,\n [

{file,"proc_lib.erl"}

,

{line,239}

]}]}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1404433902422, u'type': u'info'}
[2014-07-03 17:31:51,852] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1404433902240, u'type': u'info'}

[2014-07-03 17:31:51,852] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.121.199\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1404433901363, u'type': u'info'}

[2014-07-03 17:31:51,853] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1404433900963, u'type': u'info'}

[2014-07-03 17:31:51,853] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.121.199','ns_1@10.3.3.126',\n 'ns_1@10.3.5.11'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1404433900867, u'type': u'info'}

[2014-07-03 17:31:51,853] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 0, u'text': u"Haven't heard from a higher priority node or a master, so I'm taking over.", u'shortText': u'message', u'module': u'mb_master', u'tstamp': 1404433900699, u'type': u'info'}

[2014-07-03 17:31:51,853] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 3, u'text': u'Node ns_1@10.3.121.199 joined cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1404433890762, u'type': u'info'}

[2014-07-03 17:31:51,854] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@10.3.121.199\'. Version: "3.0.0-918-rel-enterprise".', u'shortText': u'web start ok', u'module': u'menelaus_sup', u'tstamp': 1404433890737, u'type': u'info'}

[2014-07-03 17:31:51,854] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.121.199', u'code': 4, u'text': u"Node 'ns_1@10.3.121.199' saw that node 'ns_1@10.3.5.11' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1404433890636, u'type': u'info'}

[2014-07-03 17:31:51,854] - [rest_client:2000] ERROR -

{u'node': u'ns_1@10.3.5.11', u'code': 4, u'text': u"Node 'ns_1@10.3.5.11' saw that node 'ns_1@10.3.121.199' came up. Tags: []", u'shortText': u'node up', u'module': u'ns_node_disco', u'tstamp': 1404433890634, u'type': u'info'}

[