Details
Description
[Test]
./testrunner -i INI/xdcr.4.ini -t xdcr.upgradeXDCR.UpgradeTests.offline_cluster_upgrade,initial_version=2.0.1-170-rel,sdata=False,bucket_topology=default:1>2;bucket0:1><2,upgrade_nodes=src,post-upgrade-actions=src-rebalanceout;dest-rebalanceout;dest-create_index,upgrade_version=3.0.0-670-rel
[Test Steps]
1. Install each 4 nodes with 2.0.1-1070
2. Setup 2-2 Node SRC and Dest cluster.
3. Stat replication (default > default, uniXDCR), (bucket0 <> bucket0, bixdcr)
4. Load 1000 items on default and bucket0.
5. Perform offline upgrade on SRC node to 3.0.0-70.
6. Load 1000 items on default and bucket 0.
7. Perform rebalance out operation on Source cluster. Remove non-master node. Rebalance failed after 77%.
2 Source Node, 2 Destination Node.
Initial Version: 2.0.1-170
Upgrade Version: 3.0.0-670 (Intera-Replication=TAP, XDCR = UPR)
[Error]
2014-05-15 05:11:10,196 - root - ERROR -
- rebalance failed
2014-05-15 05:11:10,636 - root - INFO - Latest logs from UI on 10.3.2.161:
2014-05-15 05:11:10,637 - root - ERROR - {u'node': u'ns_1@10.3.2.161', u'code': 2, u'text': u"Rebalance exited with reason {unexpected_exit,\n {'EXIT',<0.16848.3>,\n {badmatch,\n [{'EXIT',\n {system_limit,\n
}}]}}}\n", u'shortText': u'message', u'serverTime': u'2014-05-15T05:06:49.958Z', u'module': u'ns_orchestrator', u'tstamp': 1400155609958, u'type': u'info'}
2014-05-15 05:11:10,638 - root - ERROR - {u'node': u'ns_1@10.3.2.161', u'code': 0, u'text': u"<0.16820.3> exited with {unexpected_exit,\n {'EXIT',<0.16848.3>,\n {badmatch,\n [{'EXIT',\n {system_limit,\n
}}]}}}", u'shortText': u'message', u'serverTime': u'2014-05-15T05:06:49.931Z', u'module': u'ns_vbucket_mover', u'tstamp': 1400155609931, u'type': u'critical'}
2014-05-15 05:11:10,638 - root - ERROR -
2014-05-15 05:11:10,639 - root - ERROR -
{u'node': u'ns_1@10.3.2.161', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@10.3.2.161\' for server shutdown', u'shortText': u'message', u'serverTime': u'2014-05-15T05:06:46.828Z', u'module': u'ns_memcached', u'tstamp': 1400155606828, u'type': u'info'}2014-05-15 05:11:10,639 - root - ERROR -
{u'node': u'ns_1@10.3.2.161', u'code': 0, u'text': u'Bucket "bucket0" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-05-15T05:04:08.677Z', u'module': u'ns_vbucket_mover', u'tstamp': 1400155448677, u'type': u'info'}2014-05-15 05:11:10,640 - root - ERROR -
{u'node': u'ns_1@10.3.4.175', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@10.3.4.175\' for deletion', u'shortText': u'message', u'serverTime': u'2014-05-15T05:04:08.545Z', u'module': u'ns_memcached', u'tstamp': 1400155448545, u'type': u'info'}2014-05-15 05:11:10,641 - root - ERROR -
{u'node': u'ns_1@10.3.2.161', u'code': 0, u'text': u'Started rebalancing bucket bucket0', u'shortText': u'message', u'serverTime': u'2014-05-15T05:04:08.350Z', u'module': u'ns_rebalancer', u'tstamp': 1400155448350, u'type': u'info'}2014-05-15 05:11:10,641 - root - ERROR -
{u'node': u'ns_1@10.3.2.161', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2014-05-15T04:58:48.194Z', u'module': u'ns_vbucket_mover', u'tstamp': 1400155128194, u'type': u'info'}2014-05-15 05:11:10,642 - root - ERROR -
{u'node': u'ns_1@10.3.2.161', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2014-05-15T04:58:47.866Z', u'module': u'ns_rebalancer', u'tstamp': 1400155127866, u'type': u'info'}2014-05-15 05:11:10,642 - root - ERROR -
{u'node': u'ns_1@10.3.2.161', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@10.3.2.161'], EjectNodes = ['ns_1@10.3.4.175'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2014-05-15T04:58:47.815Z', u'module': u'ns_orchestrator', u'tstamp': 1400155127815, u'type': u'info'}[Additional information]
[Jenkins execution]
http://qa.hq.northscale.net/job/centos_x64--104_01--XDCR_upgrade-P1/3/consoleFull
Bug is consistent: Occuring in every execution
Total issued failed because of this: 12
[Impact]
1. Rebalance operation is not possible after offline upgrade
2. All online upgrade tests are failed.
Attachments
For Gerrit Dashboard: MB-11124 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
37401,2 | MB-11124 Bump maximum number of ets tables. | master | ns_server | Status: MERGED | +2 | +1 |