Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.0.0
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
Build
5.0.1-5003-> 6.0.0-1529
Testcase
./testrunner -i /tmp/testexec.19700.ini -p get-cbcollect-info=True,disable_HTP=True,get-logs=False,stop-on-failure=False,fts_quota=512,upgrade_version=6.0.0-1529 -t upgrade.upgrade_tests.UpgradeTests.test_upgrade,items=5000,initial_version=5.0.1-5003,nodes_init=3,initialize_events=create_fts_index_query_compare,initial-services-setting=kv,fts-kv,fts-kv,fts,upgrade_services_in=same,after_events=rebalance_in-check_index_type-update_index_to_upside_down-run_fts_query_and_compare,after_upgrade_services_in=kv,fts,disable_HTP=True,upgrade_test=True,init_nodes=False,after_upgrade_index_type=upside_down,GROUP=ONLINE
Steps:
- Install 5.0.1 on 3 nodes, cluster them
- Create a bucket, load 5000k docs, create default index (upside_down)
- Swap rebalance a 6.0.0 node into the cluster.
Rebalance fails with the error below -
2018-09-20 06:01:20,526 - root - INFO - Node versions in cluster [u'5.0.1-5003-enterprise', u'5.0.1-5003-enterprise', u'5.0.1-5003-enterprise'] |
2018-09-20 06:01:20,773 - root - INFO - This is swap rebalance and we will monitor vbuckets shuffling |
2018-09-20 06:01:20,773 - root - INFO - adding node 172.23.104.186:8091 to cluster |
2018-09-20 06:01:20,773 - root - INFO - adding remote node @172.23.104.186:8091 to this cluster @172.23.104.175:8091 |
2018-09-20 06:01:25,392 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.104.175', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.104.175,ns_1@172.23.104.186,ns_1@172.23.104.173,ns_1@172.23.104.174'} |
2018-09-20 06:01:35,315 - root - INFO - rebalance operation started |
2018-09-20 06:01:35,315 - root - INFO - This is swap rebalance and we will monitor vbuckets shuffling |
2018-09-20 06:01:35,599 - root - INFO - rebalance percentage : 0.00 % |
2018-09-20 06:01:35,599 - root - INFO - Rebalance - status: running, progress: 0.00% |
2018-09-20 06:01:45,691 - root - INFO - This is swap rebalance and we will monitor vbuckets shuffling |
2018-09-20 06:01:46,021 - root - INFO - rebalance percentage : 21.11 % |
2018-09-20 06:01:46,021 - root - INFO - Rebalance - status: running, progress: 21.11% |
2018-09-20 06:01:56,109 - root - INFO - This is swap rebalance and we will monitor vbuckets shuffling |
2018-09-20 06:01:56,416 - root - INFO - rebalance percentage : 41.42 % |
2018-09-20 06:01:56,416 - root - INFO - Rebalance - status: running, progress: 41.42% |
2018-09-20 06:02:06,500 - root - INFO - This is swap rebalance and we will monitor vbuckets shuffling |
2018-09-20 06:02:06,771 - root - INFO - rebalance percentage : 50.00 % |
2018-09-20 06:02:06,771 - root - INFO - Rebalance - status: running, progress: 50.00% |
2018-09-20 06:02:16,849 - root - INFO - This is swap rebalance and we will monitor vbuckets shuffling |
2018-09-20 06:02:17,115 - root - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed |
2018-09-20 06:02:17,184 - root - INFO - Latest logs from UI on 172.23.104.175: |
2018-09-20 06:02:17,184 - root - ERROR - {u'node': u'ns_1@172.23.104.186', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,fts,\n {rebalance_failed,\n {service_error,\n <<"planner: nodeDefs.ImplVersion: 5.5.0 > version: 5.0.0">>}}}', u'shortText': u'message', u'serverTime': u'2018-09-20T06:02:11.876Z', u'module': u'ns_orchestrator', u'tstamp': 1537448531876, u'type': u'critical'} |
2018-09-20 06:02:17,184 - root - ERROR - {u'node': u'ns_1@172.23.104.175', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.104.175\' for deletion', u'shortText': u'message', u'serverTime': u'2018-09-20T06:02:11.871Z', u'module': u'ns_memcached', u'tstamp': 1537448531871, u'type': u'info'} |
2018-09-20 06:02:17,185 - root - ERROR - {u'node': u'ns_1@172.23.104.186', u'code': 0, u'text': u'Bucket "default" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2018-09-20T06:01:36.692Z', u'module': u'ns_vbucket_mover', u'tstamp': 1537448496692, u'type': u'info'} |
2018-09-20 06:02:17,185 - root - ERROR - {u'node': u'ns_1@172.23.104.186', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.104.186\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2018-09-20T06:01:35.631Z', u'module': u'ns_memcached', u'tstamp': 1537448495631, u'type': u'info'} |
2018-09-20 06:02:17,185 - root - ERROR - {u'node': u'ns_1@172.23.104.186', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2018-09-20T06:01:35.396Z', u'module': u'ns_rebalancer', u'tstamp': 1537448495396, u'type': u'info'} |
2018-09-20 06:02:17,185 - root - ERROR - {u'node': u'ns_1@172.23.104.186', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.104.186','ns_1@172.23.104.173',\n 'ns_1@172.23.104.174'], EjectNodes = ['ns_1@172.23.104.175'], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2018-09-20T06:01:35.297Z', u'module': u'ns_orchestrator', u'tstamp': 1537448495297, u'type': u'info'} |
Attaching logs from the cluster -