Details
-
Bug
-
Resolution: Resolved
-
Critical
-
Cheshire-Cat
-
7.0.0-4846-enterprise
-
Untriaged
-
Windows 64-bit
-
1
-
No
Description
Scripts to Repro
./testrunner -i /tmp/win10-bucket-ops-11111222223334444.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_after_ops,nodes_out=1,replicas=1,items=10000,GROUP=OUT;P0 incremental_rebalance_out_with_ops,replicas=2,items=0,standard_buckets=2,sasl_buckets=2,standard_bucket_priority=low:high,sasl_bucket_priority=low:high,value_size=1024,GROUP=OUT;P0
|
./testrunner -i /tmp/win10-bucket-ops-11111222223334444.ini -p get-cbcollect-info=True,get-logs=True,get-coredumps=True -t failover.AutoFailoverTests.AutoFailoverTests.test_autofailover,timeout=5,num_node_failures=1,failover_orchestrator=True,failover_action=stop_server,nodes_init=3
|
Saw this failure on the above 2 tests on window runs.
We are trying to create a cluster when rebalance fails as shown below.
[2021-04-02 01:30:10,497] - [basetestcase:2388] INFO - **** add 'admin' role to 'cbadminbucket' user ****
|
[2021-04-02 01:30:10,519] - [basetestcase:332] INFO - done initializing cluster
|
[2021-04-02 01:30:11,334] - [task:775] INFO - adding node 172.23.136.109:8091 to cluster
|
[2021-04-02 01:30:11,334] - [rest_client:1487] INFO - adding remote node @172.23.136.109:8091 to this cluster @172.23.136.103:8091
|
[2021-04-02 01:30:21,353] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds
|
[2021-04-02 01:30:21,354] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
|
[2021-04-02 01:30:36,582] - [task:775] INFO - adding node 172.23.136.111:8091 to cluster
|
[2021-04-02 01:30:36,582] - [rest_client:1487] INFO - adding remote node @172.23.136.111:8091 to this cluster @172.23.136.103:8091
|
[2021-04-02 01:30:46,601] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds
|
[2021-04-02 01:30:46,602] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
|
[2021-04-02 01:31:01,480] - [task:775] INFO - adding node 172.23.136.112:8091 to cluster
|
[2021-04-02 01:31:01,480] - [rest_client:1487] INFO - adding remote node @172.23.136.112:8091 to this cluster @172.23.136.103:8091
|
[2021-04-02 01:31:11,500] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds
|
[2021-04-02 01:31:11,500] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
|
[2021-04-02 01:31:26,303] - [task:775] INFO - adding node 172.23.136.115:8091 to cluster
|
[2021-04-02 01:31:26,303] - [rest_client:1487] INFO - adding remote node @172.23.136.115:8091 to this cluster @172.23.136.103:8091
|
[2021-04-02 01:31:36,323] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds
|
[2021-04-02 01:31:36,323] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
|
[2021-04-02 01:31:56,092] - [task:775] INFO - adding node 172.23.136.114:8091 to cluster
|
[2021-04-02 01:31:56,093] - [rest_client:1487] INFO - adding remote node @172.23.136.114:8091 to this cluster @172.23.136.103:8091
|
[2021-04-02 01:32:06,116] - [rest_client:1820] INFO - rebalance progress took 10.02 seconds
|
[2021-04-02 01:32:06,117] - [rest_client:1821] INFO - sleep for 10 seconds after rebalance...
|
[2021-04-02 01:32:24,630] - [rest_client:1714] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.136.103,ns_1@172.23.136.109,ns_1@172.23.136.111,ns_1@172.23.136.112,ns_1@172.23.136.114,ns_1@172.23.136.115', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'}
|
[2021-04-02 01:32:24,685] - [rest_client:1719] INFO - rebalance operation started
|
[2021-04-02 01:32:24,689] - [rest_client:1881] INFO - rebalance percentage : 0.00 %
|
[2021-04-02 01:32:24,689] - [task:844] INFO - Rebalance - status: running, progress: 0.00%
|
[2021-04-02 01:32:34,727] - [rest_client:1864] ERROR - {'status': 'none', 'errorMessage': 'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
|
[2021-04-02 01:32:34,747] - [rest_client:3816] INFO - Latest logs from UI on 172.23.136.103:
|
[2021-04-02 01:32:34,748] - [rest_client:3817] ERROR - {'node': 'ns_1@172.23.136.103', 'type': 'critical', 'code': 0, 'module': 'ns_orchestrator', 'tstamp': 1617352347001, 'shortText': 'message', 'text': 'Rebalance exited with reason {service_rebalance_failed,index,\n {worker_died,\n {\'EXIT\',<0.8358.15>,\n {rebalance_failed,\n {service_error,\n <<"Unable to read index layout from cluster 127.0.0.1:8091. err = Invalid couchbase-server version">>}}}}}.\nRebalance Operation Id = e7e14713eb3a0bc171c1d5cf28236f1e', 'serverTime': '2021-04-02T01:32:27.001Z'}
|
cbcollect_info attached.
Attachments
Issue Links
- mentioned in
-
Page Loading...