Details
-
Bug
-
Resolution: Fixed
-
Critical
-
3.0
-
Security Level: Public
-
Build 3.0.0-918
-
Untriaged
-
Centos 64-bit
-
-
Yes
Description
http://qa.hq.northscale.net/job/centos_x64--107_01--rebalanceXDCR-P1/10/consoleFull
[Test]
./testrunner i centos_x64107_01-rebalanceXDCR-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=bidirection,ctopology=chain,doc-ops=update-delete,doc-ops-dest=update-delete,expires=60,rebalance=source-destination,num_rebalance=1,GROUP=P1
[Test Error]
[2014-07-02 03:59:21,417] - [rest_client:1208] INFO - rebalance percentage : 83.2327519408 %
[2014-07-02 03:59:22,427] - [rest_client:1192] ERROR -
- rebalance failed
[2014-07-02 03:59:22,470] - [rest_client:1999] INFO - Latest logs from UI on 10.5.2.231:
[2014-07-02 03:59:22,470] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.5.2.231', u'code': 2, u'text': u'Rebalance exited with reason {unexpected_exit,\n {\'EXIT\',<0.15215.28>,\n {bulk_set_vbucket_state_failed,\n [{\'ns_1@10.3.5.68\',\n {\'EXIT\',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7,\n [
,\n
{line,381}]},\n {misc,\'[2014-07-02 03:59:22,472] - [rest_client:2000] ERROR - {u'node': u'ns_1@10.5.2.231', u'code': 0, u'text': u'<0.6273.28> exited with {unexpected_exit,\n {\'EXIT\',<0.15215.28>,\n {bulk_set_vbucket_state_failed,\n [{\'ns_1@10.3.5.68\',\n {\'EXIT\',\n {{badmatch,wrong_rebalancer_pid},\n [{janitor_agent,set_vbucket_state,7,\n [{file,"src/janitor_agent.erl"},{line,394}]},\n {janitor_agent,\n \'
]},\n {misc,\'parallel_map/3-fun-0\',4,\n [
,
{line,86}]}]}}}]}}}', u'shortText': u'message', u'serverTime': u'2014-07-02T03:59:20.328Z', u'module': u'ns_vbucket_mover', u'tstamp': 1404298760328, u'type': u'critical'}
[2014-07-02 03:59:22,473] - [rest_client:2000] ERROR -
[2014-07-02 03:59:22,474] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.3.5.68', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@10.3.5.68\'. Version: "3.0.0-918-rel-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2014-07-02T03:59:19.612Z', u'module': u'menelaus_sup', u'tstamp': 1404298759612, u'type': u'info'}[2014-07-02 03:59:22,475] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.3.5.68', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@10.3.5.68\' for server shutdown', u'shortText': u'message', u'serverTime': u'2014-07-02T03:59:19.443Z', u'module': u'ns_memcached', u'tstamp': 1404298759443, u'type': u'info'}[2014-07-02 03:59:22,476] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.5.2.231', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:23.446Z', u'module': u'ns_vbucket_mover', u'tstamp': 1404298583446, u'type': u'info'}[2014-07-02 03:59:22,477] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.3.5.68', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@10.3.5.68\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:22.539Z', u'module': u'ns_memcached', u'tstamp': 1404298582539, u'type': u'info'}[2014-07-02 03:59:22,478] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.5.2.231', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:21.089Z', u'module': u'ns_rebalancer', u'tstamp': 1404298581089, u'type': u'info'}[2014-07-02 03:59:22,479] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.5.2.231', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.5.2.233','ns_1@10.5.2.231',\n 'ns_1@10.5.2.232','ns_1@10.3.5.68'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:20.857Z', u'module': u'ns_orchestrator', u'tstamp': 1404298580857, u'type': u'info'}[2014-07-02 03:59:22,479] - [rest_client:2000] ERROR -
{u'node': u'ns_1@10.3.5.68', u'code': 3, u'text': u'Node ns_1@10.3.5.68 joined cluster', u'shortText': u'message', u'serverTime': u'2014-07-02T03:56:20.807Z', u'module': u'ns_cluster', u'tstamp': 1404298580807, u'type': u'info'}[Operation failed]
adding remote node @10.3.5.68:8091 to this cluster @10.5.2.231:8091