Details
Description
http://qa.hq.northscale.net/job/ubuntu-32-2.0-swaprebalance-test-P0/67/consoleFull
/testrunner -i /tmp/swaprebalance-32.ini get-cbcollect-info=True,GROUP=P0 -t swaprebalance.SwapRebalanceBasicTests.do_test,replica=2,num-buckets=4,num-swap=1,swap-orchestrator=True,GROUP=P0
[2013-05-02 07:08:21,444] - [rest_client:900] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.2.154&user=Administrator&knownNodes=ns_1%4010.3.2.157%2Cns_1%4010.3.2.154%2Cns_1%4010.3.2.156%2Cns_1%4010.3.2.153
[2013-05-02 07:45:28,866] - [rest_client:989] ERROR -
{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed
[2013-05-02 07:45:28,866] - [rest_client:990] INFO - Latest logs from UI:
[2013-05-02 07:45:28,998] - [rest_client:991] ERROR - {u'node': u'ns_1@10.3.2.154', u'code': 2, u'text': u'Rebalance exited with reason {badmatch,\n [{ok,{99,1,<<"does_not_exist">>}},\n {ok,{99,1,<<"does_not_exist">>}}]}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1367505935470, u'type': u'info'}
[2013-05-02 07:45:28,999] - [rest_client:991] ERROR -
[2013-05-02 07:45:28,999] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.154', u'code': 2, u'text': u'Shutting down bucket "bucket-2" on \'ns_1@10.3.2.154\' for deletion', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1367505738158, u'type': u'info'}[2013-05-02 07:45:28,999] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.153', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.2.153\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1367505737666, u'type': u'info'}[2013-05-02 07:45:28,999] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.154', u'code': 0, u'text': u'Started rebalancing bucket bucket-1', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1367505737492, u'type': u'info'}[2013-05-02 07:45:29,000] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.154', u'code': 0, u'text': u'Bucket "bucket-2" rebalance appears to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1367504758288, u'type': u'info'}[2013-05-02 07:45:29,000] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.154', u'code': 2, u'text': u'Shutting down bucket "bucket-3" on \'ns_1@10.3.2.154\' for deletion', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1367504757414, u'type': u'info'}[2013-05-02 07:45:29,000] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.153', u'code': 1, u'text': u'Bucket "bucket-2" loaded on node \'ns_1@10.3.2.153\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1367504756954, u'type': u'info'}[2013-05-02 07:45:29,001] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.154', u'code': 0, u'text': u'Started rebalancing bucket bucket-2', u'shortText': u'message', u'module': u'ns_rebalancer', u'tstamp': 1367504756794, u'type': u'info'}[2013-05-02 07:45:29,001] - [rest_client:991] ERROR -
{u'node': u'ns_1@10.3.2.154', u'code': 0, u'text': u'Bucket "bucket-3" rebalance appears to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1367503711715, u'type': u'info'}ERROR