Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.2.1
-
Enterprise Edition 7.2.1 build 5928
-
Untriaged
-
Centos 64-bit
-
0
-
Yes
Description
Script to Repro
./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.2.1-5928 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true
|
It failed in the following step which is indexing swap rebalance. We have not seen this before on any run in 7.2.1 (probably something transient or probably environmental?)
[2023-08-26T04:38:27-07:00, sequoiatools/couchbase-cli:7.1:3fb73c] server-add -c 172.23.108.139:8091 --server-add https://172.23.108.132 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
|
[2023-08-26T04:38:41-07:00, sequoiatools/couchbase-cli:7.1:beb53b] rebalance -c 172.23.108.139:8091 --server-remove 172.23.105.0 -u Administrator -p password
|
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.108.139:8091 --server-remove 172.23.105.0 -u Administrator -p password]
|
|
docker logs beb53b
|
docker start beb53b
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
+debug.log on 172.23.108.139 +
[error_logger:error,2023-08-26T09:35:20.204-07:00,ns_1@172.23.108.139:<0.8815.559>:ale_error_logger_handler:do_log:101]
|
=========================CRASH REPORT=========================
|
crasher:
|
initial call: erlang:apply/2
|
pid: <0.8815.559>
|
registered_name: []
|
exception error: no match of right hand side value
|
{leader_activities_error,
|
{default,rebalance},
|
{quorum_lost,{lease_lost,'ns_1@172.23.105.210'}}}
|
in function ns_rebalancer:rebalance/5 (src/ns_rebalancer.erl, line 453)
|
ancestors: [<0.29510.0>,ns_orchestrator_child_sup,ns_orchestrator_sup,
|
mb_master_sup,mb_master,leader_registry_sup,
|
leader_services_sup,<0.28076.0>,ns_server_sup,
|
ns_server_nodes_sup,<0.269.0>,ns_server_cluster_sup,
|
root_sup,<0.144.0>]
|
message_queue_len: 0
|
messages: []
|
links: [<0.29510.0>]
|
dictionary: []
|
trap_exit: false
|
status: running
|
heap_size: 75113
|
stack_size: 29
|
reductions: 12728
|
neighbours:
|
|
[user:error,2023-08-26T09:35:20.206-07:00,ns_1@172.23.108.139:<0.29510.0>:ns_orchestrator:log_rebalance_completion:1433]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,rebalance},
|
{quorum_lost,
|
{lease_lost,'ns_1@172.23.105.210'}}}},
|
[{ns_rebalancer,rebalance,5,
|
[{file,"src/ns_rebalancer.erl"},{line,453}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,226}]}]}.
|
Rebalance Operation Id = 972a85c1bf48db78a37b2406713d20b9
|
[ns_server:debug,2023-08-26T09:35:20.207-07:00,ns_1@172.23.108.139:<0.29510.0>:auto_rebalance:retry_rebalance:58]Retry rebalance is not enabled. Failed Rebalance with Id 972a85c1bf48db78a37b2406713d20b9 will not be retried.
|
[ns_server:debug,2023-08-26T09:35:20.248-07:00,ns_1@172.23.108.139:chronicle_kv_log<0.27844.0>:chronicle_kv_log:log:59]update (key: counters, rev: {<<"9acaa69afb3a3d177cb494198d8d8bf9">>,89299})
|
[{rebalance_fail,{1693067720,1}},
|
{rebalance_start,{1693049922,12}},
|
{rebalance_success,{1693049432,11}}]
|
cbcollect_info attached. Last successful run was on 7.2.1-5924.