Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.1.0-2211
Test : -test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
Scale : 3
Iteration : 2nd
Swap rebalance of indexer nodes failed with error
[2022-02-02T18:04:21-08:00, sequoiatools/couchbase-cli:7.1:8b7f33] server-add -c 172.23.108.103:8091 --server-add https://172.23.123.28 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
|
[2022-02-02T18:04:34-08:00, sequoiatools/couchbase-cli:7.1:35ccb6] rebalance -c 172.23.108.103:8091 --server-remove 172.23.104.70 -u Administrator -p password
|
[ns_server:error,2022-02-02T18:07:11.460-08:00,ns_1@172.23.108.103:service_rebalancer-index<0.27778.825>:service_rebalancer:run_rebalance_worker:119]Worker terminated abnormally: {'EXIT',<0.28115.825>,
|
{rebalance_failed,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/dropIndex\": EOF">>}}}
|
[user:error,2022-02-02T18:07:11.463-08:00,ns_1@172.23.108.103:<0.22228.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,index,
|
{worker_died,
|
{'EXIT',<0.28115.825>,
|
{rebalance_failed,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/dropIndex\": EOF">>}}}}}.
|
Rebalance Operation Id = 09488d5d66cb7ecf30efdf6bcbeea817
|
Following can be seen in the indexer logs of 172.23.96.251
2022-02-02T18:07:11.476-08:00 [Info] RebalanceServiceManager::cleanupTransferTokens Cleaning Up TransferTokene4:70:34:b9:28:34:6c:da MasterId: 1646a68698496726ca6b4f619e7281e4 SourceId: 50b128ec16cc5f3930c1dba10389e169 (172.23.96.252:8091) DestId: 1eef267084b99187a70656a476c815f1 (172.23.120.245:8091) RebalId: 74f491857bad74d6ed017c3b93936e9b State: TransferTokenReady BuildSource: Dcp TransferMode: Move Error: Post "http://127.0.0.1:9102/dropIndex": EOF InstId: 16657099552060681481 RealInstId: 3989044029751808803 Partitions: [5] Versions: [2] Inst:
|
And the following can be seen in the logs of 172.23.96.252
2022-02-02T18:07:11.375-08:00 [Info] Rebalancer::queueDropIndexLOCKED: Queued index idx8_jHlyb22z for drop, ttid: TransferTokene4:70:34:b9:28:34:6c:da
|
2022-02-02T18:07:11.375-08:00 [Info] Rebalancer::processDropIndexQueue processing drop index request for ttid: TransferTokene4:70:34:b9:28:34:6c:da
|
2022-02-02T18:07:11.377-08:00 [Error] Rebalancer::dropIndexWhenIdle: Error drop index on 127.0.0.1:9102/dropIndex Post "http://127.0.0.1:9102/dropIndex": EOF
|