Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.1.0
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.1.0-2375
Test : -test tests/integration/neo/test_neo_magma_milestone4.yml -scope tests/integration/neo/scope_neo_magma.yml
Scale : 3
Iteration : 1st
Rebalance operation to add 2 data nodes to the cluster failed due to index service error.
[2022-02-24T17:18:43-08:00, sequoiatools/couchbase-cli:7.1:affc1f] server-add -c 172.23.108.139:8091 --server-add https://172.23.108.141 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2022-02-24T17:19:12-08:00, sequoiatools/couchbase-cli:7.1:e1dc65] server-add -c 172.23.108.139:8091 --server-add https://172.23.108.146 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
|
[2022-02-24T17:19:25-08:00, sequoiatools/couchbase-cli:7.1:29f1c7] rebalance -c 172.23.108.139:8091 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.108.139:8091 -u Administrator -p password]
|
|
|
docker logs 29f1c7
|
docker start 29f1c7
|
|
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[2022-02-24T18:05:26-08:00, sequoiatools/cmd:3c2d00] 60
|
Following error is seen in the error.log of the orchestrator node 172.23.108.139 :
[ns_server:error,2022-02-24T18:05:18.154-08:00,ns_1@172.23.108.139:service_rebalancer-index<0.11108.364>:service_rebalancer:run_rebalance_worker:119]Worker terminated abnormally: {'EXIT',<0.11252.364>,
|
{rebalance_failed,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/createIndexRebalance\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}
|
[user:error,2022-02-24T18:05:18.158-08:00,ns_1@172.23.108.139:<0.26408.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,index,
|
{worker_died,
|
{'EXIT',<0.11252.364>,
|
{rebalance_failed,
|
{service_error,
|
<<"Post \"http://127.0.0.1:9102/createIndexRebalance\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}}}.
|
Rebalance Operation Id = 23c0e2e34dc0273a87696f4c02caa654
|
Index nodes : 172.23.104.249:8091 172.23.105.0:8091 172.23.105.39:8091 172.23.106.54:8091 172.23.108.132:8091 172.23.108.140:8091
Following can be seen in the indexer.log of 172.23.108.132 :
2022-02-24T18:05:13.149-08:00 [Info] Rebalancer::computeProgress 0.5333333333333333
|
2022-02-24T18:05:13.149-08:00 [Info] RebalanceServiceManager::GetCurrentTopology returns &{Rev:[0 0 0 0 0 0 1 176] Nodes:[c5efaf717f41d269233722f69e52c58e 2bce64bb204b0dcf2f57b1d3bc65321d be6ed18f0a850f9ad04d4eea8c477ed3 d829e79937005390d34fe2b321f8069b c1ccb0209b211757c6bb9cdada80602f e6a3ca2a26e951a04fcb320c91e9eed9] IsBalanced:true Messages:[]}
|
2022-02-24T18:05:13.149-08:00 [Info] RebalanceServiceManager::GetTaskList returns &{Rev:[0 0 0 0 0 0 1 176] Tasks:[{Rev:[0 0 0 0 0 0 0 0] ID:prepare/659eb65e1c9de05b25c6a9a392302b0e Type:task-prepared Status:task-running IsCancelable:true Progress:0 DetailedProgress:map[] Description: ErrorMessage: Extra:map[rebalanceId:659eb65e1c9de05b25c6a9a392302b0e]} {Rev:[0 0 0 0 0 0 1 171] ID:rebalance/659eb65e1c9de05b25c6a9a392302b0e Type:task-rebalance Status:task-running IsCancelable:true Progress:0.5333333333333333 DetailedProgress:map[] Description: ErrorMessage: Extra:map[rebalanceId:659eb65e1c9de05b25c6a9a392302b0e]}]}
|
2022-02-24T18:05:13.150-08:00 [Info] RebalanceServiceManager::GetTaskList [0 0 0 0 0 0 1 176]
|
2022-02-24T18:05:13.151-08:00 [Info] RebalanceServiceManager::GetCurrentTopology [0 0 0 0 0 0 1 176]
|
2022-02-24T18:05:16.075-08:00 [Info] Rebalancer::decodeTransferToken TransferToken TransferToken97:bc:0:a:97:93:68:d1 MasterId: c1ccb0209b211757c6bb9cdada80602f SourceId: c5efaf717f41d269233722f69e52c58e (172.23.104.249:8091) DestId: be6ed18f0a850f9ad04d4eea8c477ed3 (172.23.105.39:8091) RebalId: 659eb65e1c9de05b25c6a9a392302b0e State: TransferTokenCreated BuildSource: Dcp TransferMode: Move Error: Post "http://127.0.0.1:9102/createIndexRebalance": context deadline exceeded (Client.Timeout exceeded while awaiting headers) InstId: 7478682641045126922 RealInstId: 1830343518766723011 Partitions: [1] Versions: [1] Inst:
|
InstId: 1830343518766723011
|
Defn: DefnId: 17548950556193135594 Name: idx2_KPofN5aK Using: plasma Bucket: bucket7 Scope/Id: scope_1/9 Collection/Id: coll_4/11 IsPrimary: false NumReplica: 3 InstVersion: 1
|
SecExprs: <ud>([`free_breakfast` `type` `free_parking` array_count(`public_likes`) `price` `country`])</ud>
|
Desc: [false false false false false false]
|
PartitionScheme: KEY
|
HashScheme: CRC32 PartitionKeys: [(meta().`id`)] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
|
State: INDEX_STATE_ACTIVE
|
RState: RebalActive
|
Stream: NIL_STREAM
|
Version: 0
|
ReplicaId: 2
|
PartitionContainer: <nil>
|
2022-02-24T18:05:16.077-08:00 [Error] Rebalancer::processTokenAsMaster Detected TransferToken in Error state MasterId: c1ccb0209b211757c6bb9cdada80602f SourceId: c5efaf717f41d269233722f69e52c58e (172.23.104.249:8091) DestId: be6ed18f0a850f9ad04d4eea8c477ed3 (172.23.105.39:8091) RebalId: 659eb65e1c9de05b25c6a9a392302b0e State: TransferTokenCreated BuildSource: Dcp TransferMode: Move Error: Post "http://127.0.0.1:9102/createIndexRebalance": context deadline exceeded (Client.Timeout exceeded while awaiting headers) InstId: 7478682641045126922 RealInstId: 1830343518766723011 Partitions: [1] Versions: [1] Inst:
|
InstId: 1830343518766723011
|
Defn: DefnId: 17548950556193135594 Name: idx2_KPofN5aK Using: plasma Bucket: bucket7 Scope/Id: scope_1/9 Collection/Id: coll_4/11 IsPrimary: false NumReplica: 3 InstVersion: 1
|
SecExprs: <ud>([`free_breakfast` `type` `free_parking` array_count(`public_likes`) `price` `country`])</ud>
|
Desc: [false false false false false false]
|
PartitionScheme: KEY
|
HashScheme: CRC32 PartitionKeys: [(meta().`id`)] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
|
State: INDEX_STATE_ACTIVE
|
RState: RebalActive
|
Stream: NIL_STREAM
|
Version: 0
|
ReplicaId: 2
|
PartitionContainer: <nil>
|
. Abort.
|
2022-02-24T18:05:16.077-08:00 [Info] Rebalancer::doFinish Cleanup Post "http://127.0.0.1:9102/createIndexRebalance": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
|
2022-02-24T18:05:16.078-08:00 [Info] Rebalancer::observeRebalance exiting err <nil>
|
2022-02-24T18:05:16.078-08:00 [Info] Rebalancer::getNodeIndexerStatsLoop: Done received
|