Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
7.1.0-2506
-
Untriaged
-
Windows 64-bit
-
1
-
Yes
-
CX Sprint 285
Description
This test last passed in build 7.1.0-2475
Steps to reproduce -
1. Have a 5 node cluster with 1 KV and 4 CBAS nodes.
2. Load bucket, scopes and collection on KV and load data into them.
3. Create dataverses, datasets, indexes on CBAS.
4. Set CBAS replica to 3 and rebalance so that replica can take effect.
5. Now add 1 more CBAS node and rebalance.
6. Rebalance fails.
2022-03-23 00:22:00,713 | test | ERROR | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1541] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'162b9d7f810e5663b4ec3c930cfe4406', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=46d10028050de2693d07720d448c86c1', u'status': u'notRunning'} - rebalance failed
|
2022-03-23 00:22:01,009 | test | INFO | pool-3-thread-8 | [rest_client:print_UI_logs:2706] Latest logs from UI on 172.23.136.160:
|
2022-03-23 00:22:01,009 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.160', u'tstamp': 1648020114468L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:54.468Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,cbas,\n {worker_died,\n {\'EXIT\',<0.13004.0>,\n {rebalance_failed,\n {service_error,\n <<"Rebalance 01b142cc9284e45d0b9837d138c44468 failed: java.lang.Exception: replica 6@172.23.136.161:9120 failed">>}}}}}.\nRebalance Operation Id = cb5b43bcdb429ad048c19a57c1af5703'}
|
2022-03-23 00:22:01,009 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'warning', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020113771L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:53.771Z', u'text': u"Analytics Service unable to successfully rebalance 01b142cc9284e45d0b9837d138c44468 due to 'java.lang.Exception: replica 6@172.23.136.161:9120 failed'; see analytics_info.log for details"}
|
2022-03-23 00:22:01,010 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020107065L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:47.065Z', u'text': u'Analytics collection wZGmSDHQTbUtuCLeVJ7Z0O.yjPl9I1mlsPEAurWshziaULJedlT rebalanced. Rebalance progress now is 0.99. Remaining analytics collections: 0/15'}
|
2022-03-23 00:22:01,010 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020106545L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:46.545Z', u'text': u'Analytics collection wZGmSDHQTbUtuCLeVJ7Z0O.g7mrPc2trqmmR2YcGSISLKmnj rebalanced. Rebalance progress now is 0.9071196105150946. Remaining analytics collections: 1/15'}
|
2022-03-23 00:22:01,012 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020106014L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:46.014Z', u'text': u'Analytics collection wZGmSDHQTbUtuCLeVJ7Z0O.R6eb7KvKgTNrk2d9k59lzKM32HGGKi rebalanced. Rebalance progress now is 0.8493139085001821. Remaining analytics collections: 2/15'}
|
2022-03-23 00:22:01,013 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020105656L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:45.656Z', u'text': u'Analytics collection wZGmSDHQTbUtuCLeVJ7Z0O.K28ljoZ rebalanced. Rebalance progress now is 0.7788225558835925. Remaining analytics collections: 3/15'}
|
2022-03-23 00:22:01,013 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020105110L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:45.110Z', u'text': u'Analytics collection wZGmSDHQTbUtuCLeVJ7Z0O.CL6Ddt8puyZL rebalanced. Rebalance progress now is 0.7210219770511136. Remaining analytics collections: 4/15'}
|
2022-03-23 00:22:01,013 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020104180L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:44.180Z', u'text': u'Analytics collection JsHnB0Qsp9qvZsZOig6p.zjXxUzTUYogzecQ rebalanced. Rebalance progress now is 0.6057582565373447. Remaining analytics collections: 5/15'}
|
2022-03-23 00:22:01,015 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020103646L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:43.646Z', u'text': u'Analytics collection JsHnB0Qsp9qvZsZOig6p.zPXuXAW2i9y rebalanced. Rebalance progress now is 0.5479568378388932. Remaining analytics collections: 6/15'}
|
2022-03-23 00:22:01,015 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2708] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.136.161', u'tstamp': 1648020103280L, u'shortText': u'message', u'serverTime': u'2022-03-23T00:21:43.280Z', u'text': u'Analytics collection JsHnB0Qsp9qvZsZOig6p.p3xRp0zoV0niIyzrsNcR9 rebalanced. Rebalance progress now is 0.47746817804257863. Remaining analytics collections: 7/15'}
|
2022-03-23 00:22:01,016 | test | ERROR | pool-3-thread-8 | [task:call:326] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'162b9d7f810e5663b4ec3c930cfe4406', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=46d10028050de2693d07720d448c86c1', u'status': u'notRunning'} - rebalance failed
|
Test Link - http://qa.sc.couchbase.com/job/test_suite_executor-TAF/175788/console
Same test works perfectly fine on Centos.
Attachments
Issue Links
- relates to
-
-
- Closed
-