Details
Critical Description
Upgrade Couchbase Server from 5.0.0-3519 to 5.5.0-2211.
Add new version nodes to cluser. Rebalance passed.
Then remove old version out of cluster.
The upgrade failed when remove old version (5.0.0) nodes (172.23.98.71, 172.23.98.72, 172.23.98.73, 172.23.98.74') out of cluster with errors:
2018-03-20 01:43:25,802] - [rest_client:1569] ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
|
[2018-03-20 01:43:25,834] - [rest_client:3102] INFO - Latest logs from UI on 172.23.98.71:
|
[2018-03-20 01:43:25,835] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.98.75', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,index,\n {rebalance_failed,\n {service_error,\n <<"Post http://127.0.0.1:9102/dropIndex: net/http: request canceled (Client.Timeout exceeded while awaiting headers)">>}}}', u'shortText': u'message', u'serverTime': u'2018-03-20T01:43:18.747Z', u'module': u'ns_orchestrator', u'tstamp': 1521535398747, u'type': u'critical'}
|
[2018-03-20 01:43:25,835] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.98.71', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.98.71\' for deletion', u'shortText': u'message', u'serverTime': u'2018-03-20T01:41:04.626Z', u'module': u'ns_memcached', u'tstamp': 1521535264626, u'type': u'info'}
|
[2018-03-20 01:43:25,836] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.98.72', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.98.72\' for deletion', u'shortText': u'message', u'serverTime': u'2018-03-20T01:41:04.602Z', u'module': u'ns_memcached', u'tstamp': 1521535264602, u'type': u'info'}
|
[2018-03-20 01:43:25,836] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.98.75', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-03-20T01:39:55.059Z', u'module': u'ns_vbucket_mover', u'tstamp': 1521535195059, u'type': u'info'}
|
[2018-03-20 01:43:25,837] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.98.75', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2018-03-20T01:39:54.641Z', u'module': u'ns_rebalancer', u'tstamp': 1521535194641, u'type': u'info'}
|
[2018-03-20 01:43:25,837] - [rest_client:3103] ERROR - {u'node': u'ns_1@172.23.98.75', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.98.76','ns_1@172.23.98.78',\n 'ns_1@172.23.98.75','ns_1@172.23.98.77'], EjectNodes = ['ns_1@172.23.98.71',\n 'ns_1@172.23.98.72',\n
|
Test to run:
./testrunner -i /tmp/centos6_upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.test_swap_rebalance_with_services,initial_version=5.0.0-3519,nodes_init=4,swap_num_servers=4,ddocs_num=3,pre_upgrade=create_index,during_upgrade=,post_upgrade=create_index,doc-per-day=1,dataset=default,groups=simple,upgrade_version=5.5.0-2211,skip_init_check_cbserver=True,groups=simple,gsi_type=memory_optimized
|
I will check the last stable build for this test.