Details
-
Bug
-
Resolution: Duplicate
-
Blocker
-
3.0
-
Security Level: Public
-
None
-
Build 3.0.0-973
-
Untriaged
-
Centos 64-bit
-
-
Unknown
Description
http://qa.hq.northscale.net/job/centos_x64--107_01--rebalanceXDCR-P1/27/consoleFull
[Test]
./testrunner i centos_x64107_01-rebalanceXDCR-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=bidirection,ctopology=chain,doc-ops=update-delete,doc-ops-dest=update-delete,rebalance=source-destination,num_rebalance=1,GROUP=P0
[Test Error]
[2014-07-17 06:16:20,851] - [rest_client:1200] ERROR -
- rebalance failed
[2014-07-17 06:16:20,880] - [rest_client:2011] INFO - Latest logs from UI on 10.5.2.228:
[2014-07-17 06:16:20,880] - [rest_client:2012] ERROR -
[2014-07-17 06:16:20,881] - [rest_client:2012] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n {\'EXIT\',<0.3547.37>,\n {wait_seqno_persisted_failed,"default",272,\n 216,\n [{\'ns_1@10.5.2.234\',\n {\'EXIT\',\n {{{noproc,\n {gen_server,call,\n [\'upr_replicator-default-ns_1@10.5.2.229\',\n
{takeover,615},\n infinity]}},\n {gen_server,call,\n [\'replication_manager-default\',\n {upr_takeover,\'ns_1@10.5.2.229\',615},\n infinity]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@10.5.2.234\'},\n {if_rebalance,<0.4419.36>,\n {wait_seqno_persisted,272,216}},\n infinity]}}}}]}}}\n', u'shortText': u'message', u'serverTime': u'2014-07-17T06:16:32.008Z', u'module': u'ns_orchestrator', u'tstamp': 1405602992008, u'type': u'info'}[2014-07-17 06:16:20,881] - [rest_client:2012] ERROR - {u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'<0.2966.37> exited with {unexpected_exit,\n {\'EXIT\',<0.3547.37>,\n {wait_seqno_persisted_failed,"default",272,216,\n [{\'ns_1@10.5.2.234\',\n {\'EXIT\',\n {{{noproc,\n {gen_server,call,\n [\'upr_replicator-default-ns_1@10.5.2.229\',\n {takeover,615}
,\n infinity]}},\n {gen_server,call,\n [\'replication_manager-default\',\n
{upr_takeover,\'ns_1@10.5.2.229\',615},\n infinity]}},\n {gen_server,call,\n [
{\'janitor_agent-default\',\'ns_1@10.5.2.234\'},\n {if_rebalance,<0.4419.36>,\n {wait_seqno_persisted,272,216}},\n infinity]}}}}]}}}', u'shortText': u'message', u'serverTime': u'2014-07-17T06:16:31.990Z', u'module': u'ns_vbucket_mover', u'tstamp': 1405602991990, u'type': u'critical'}
[2014-07-17 06:16:20,882] - [rest_client:2012] ERROR - {u'node': u'ns_1@10.5.2.229', u'code': 0, u'text': u"Control connection to memcached on 'ns_1@10.5.2.229' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'serverTime': u'2014-07-17T06:16:31.802Z', u'module': u'ns_memcached', u'tstamp': 1405602991802, u'type': u'info'}
[2014-07-17 06:16:20,883] - [rest_client:2012] ERROR -
[2014-07-17 06:16:20,883] - [rest_client:2012] ERROR -
{u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-07-17T06:12:36.782Z', u'module': u'ns_vbucket_mover', u'tstamp': 1405602756782, u'type': u'info'}[2014-07-17 06:16:20,884] - [rest_client:2012] ERROR -
{u'node': u'ns_1@10.5.2.234', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.5.2.234\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-07-17T06:12:36.447Z', u'module': u'ns_memcached', u'tstamp': 1405602756447, u'type': u'info'}[2014-07-17 06:16:20,885] - [rest_client:2012] ERROR -
{u'node': u'ns_1@10.5.2.228', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2014-07-17T06:12:35.433Z', u'module': u'ns_rebalancer', u'tstamp': 1405602755433, u'type': u'info'}[2014-07-17 06:16:20,885] - [rest_client:2012] ERROR -
{u'node': u'ns_1@10.5.2.228', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.5.2.230','ns_1@10.5.2.229',\n 'ns_1@10.5.2.234','ns_1@10.5.2.228'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2014-07-17T06:12:35.303Z', u'module': u'ns_orchestrator', u'tstamp': 1405602755303, u'type': u'info'}[2014-07-17 06:16:20,886] - [rest_client:2012] ERROR -
{u'node': u'ns_1@10.5.2.234', u'code': 3, u'text': u'Node ns_1@10.5.2.234 joined cluster', u'shortText': u'message', u'serverTime': u'2014-07-17T06:12:35.245Z', u'module': u'ns_cluster', u'tstamp': 1405602755245, u'type': u'info'}ERROR