Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
7.6.0-1970
-
Untriaged
-
0
-
Unknown
Description
In GSI 5k system test, during the scale out phase, rebalance failed with error and is stuck in the re-attempt of rebalance
Test run logs are available here -
http://qe-jenkins1.sc.couchbase.com/job/cp-cli-gsi-system-test-2/7/console
http://qe-jenkins1.sc.couchbase.com/job/cp-cli-gsi-system-test-2/6/console
[ns_server:error,2024-01-02T13:51:54.381Z,ns_1@svc-q-node-008.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com:service_manager-index<0.12725.931>:service_manager:run_op_worker:230]Agent terminated during op: rebalance, {'DOWN', |
#Ref<0.1541577234.162529288.97074>, |
process,<37040.4683.0>, |
{lost_connection,
|
{'ns_1@svc-i-node-011.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com', |
shutdown}}}
|
[user:error,2024-01-02T13:51:54.383Z,ns_1@svc-q-node-008.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com:<0.3377.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{agent_died,<37040.4683.0>, |
{lost_connection,
|
{'ns_1@svc-i-node-011.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com', |
shutdown}}}}.
|
Rebalance Operation Id = 13d27c6da92a60a3ce6a77f8b77e38fd
|
[ns_server:error,2024-01-02T13:52:41.969Z,ns_1@svc-q-node-008.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com:service_manager-index-worker<0.13592.932>:service_agent:process_bad_results:990]Service call prepare_rebalance (service index) failed on some nodes: |
[{'ns_1@svc-i-node-011.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com', |
{error,
|
{unknown_error,
|
<<"indexer rebalance failure - cleanup pending from previous failed/aborted rebalance/failover/move index. please retry the request later.">>}}}] |
[ns_server:error,2024-01-02T13:52:41.970Z,ns_1@svc-q-node-008.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com:service_manager-index<0.16377.932>:service_manager:run_op_worker:219]Worker terminated abnormally: {'EXIT',<0.13592.932>, |
{{badmatch,
|
{error,
|
{bad_nodes,index,prepare_rebalance,
|
[{'ns_1@svc-i-node-011.fmnmlegfrcqhzlkq.sandbox.nonprod-project-avengers.com', |
{error,
|
{unknown_error,
|
<<"indexer rebalance failure - cleanup pending from previous failed/aborted rebalance/failover/move index. please retry the request later.">>}}}]}}}, |
[{service_manager,rebalance_op,5, |
[{file,"src/service_manager.erl"}, |
{line,338}]}, |
{service_manager,do_run_op,1, |
[{file,"src/service_manager.erl"}, |
{line,257}]}, |
{proc_lib,init_p,3, |
[{file,"proc_lib.erl"},{line,225}]}]}} |
Logs are available here -