Critical Description
During the scaling out operation of the indexer nodes the below rebalance failure appears transiently before eventually succeeding(in capella environment on AWS machines)
Rebalance exited with reason {service_rebalance_failed,index,
|
{agent_died,<36982.3326.0>,noconnection}}. |
Rebalance Operation Id = f7872485ea18986202c876adbd644689
|
Steps to repro
- Initialise a 8 node cluster with following config - kv:n1ql-kv:n1ql-kv:n1ql-index-index-index-index-index
- Create a bucket with scopes and collection and load docs and create indexes
- Trigger a scale out operation wherein two indexer nodes are added to the cluster
- During this operation it is observed that the rebalance fails transiently
Logs -
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-dq-node-001.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-dq-node-002.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-dq-node-003.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-004.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-005.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-006.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-007.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-008.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-009.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/yash-test/collectinfo-2024-02-12T112607-ns_1%40svc-i-node-010.k6n16c70v7uyqsyl.sandbox.nonprod-project-avengers.com.zip
