Details
-
Bug
-
Resolution: Unresolved
-
Major
-
7.6.0
-
7.6.0-2095
-
Untriaged
-
0
-
Unknown
Description
It appears that the rebalance has failed with the following message -
[ns_server:error,2024-02-06T12:48:26.585-08:00,ns_1@172.23.218.84:service_manager-index<0.26641.166>:service_manager:run_op_worker:219]Worker terminated abnormally: {'EXIT',<0.26804.166>, |
{task_failed,rebalance,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/dropIndex\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}} |
[ns_server:info,2024-02-06T12:48:26.587-08:00,ns_1@172.23.218.84:rebalance_agent<0.29522.0>:rebalance_agent:handle_down:290]Rebalancer process <0.24860.166> died (reason {service_rebalance_failed, |
index,
|
{worker_died,
|
{'EXIT',<0.26804.166>, |
{task_failed,rebalance,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/dropIndex\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}}}). |
[user:error,2024-02-06T12:48:26.591-08:00,ns_1@172.23.218.84:<0.30057.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index, |
{worker_died,
|
{'EXIT',<0.26804.166>, |
{task_failed,rebalance,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/dropIndex\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}}}. |
Seen during iteration 12. Auto-failover was enabled about 20 minutes prior to the rebalance failure and the test triggered an autofailover by stop couchbase server on node 218.87
[2024-02-06T12:20:37-08:00, sequoiatools/cmd:6bc9fe] 300 |
[2024-02-06T12:26:26-08:00, sequoiatools/couchbase-cli:7.6:29d1dc] setting-autofailover -c 172.23.218.86:8091 -u Administrator -p password --enable-auto-failover=1 --auto-failover-timeout=30 --max-failovers=1 |
[2024-02-06T12:26:32-08:00, sequoiatools/cmd:08e601] 10 |
[2024-02-06T12:26:48-08:00, sequoiatools/cbutil:d93355] /cbinit.py 172.23.218.87 root couchbase stop |
[2024-02-06T12:26:54-08:00, sequoiatools/cmd:353888] 40 |
[2024-02-06T12:27:40-08:00, sequoiatools/couchbase-cli:7.6:53f970] rebalance -c 172.23.218.86:8091 -u Administrator -p password |
→
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.6:[rebalance -c 172.23.218.86:8091 -u Administrator |
Cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.120.202.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.83.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.84.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.85.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.87.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.89.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.96.207.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.96.231.zip