[System Test on cloud] Fatal error MetakvSet Failed to set /indexing/rebalance/ShardToken2

While analysing a bunch of rebalance failures during the provisioned system test run, these errors were seen on one of the indexer nodes -

Seen svc-i-node-013.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com

2024-02-14T16:42:58.608+00:00 [Fatal] MetakvSet Failed to set /indexing/rebalance/ShardToken2:78:43:e1:36:60:2f:3b: Put "http://127.0.0.1:8091/_metakv/indexing/rebalance/ShardToken2:78:43:e1:36:60:2f:3b": dial tcp 127.0.0.1:8091: connect: connection refused

This error might have caused a series of rebalance failures (Unsure if that is the reason).

Note: This is a test on cloud, so the test has not induced any errors.

cbcollect ->

https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-d-node-002.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-d-node-003.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-d-node-015.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-d-node-016.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-004.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-005.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-006.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-007.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-008.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-009.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-i-node-013.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-q-node-010.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-q-node-011.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestCapella/collectinfo-2024-02-14T173417-ns_1%40svc-q-node-014.knjzvqrr6w6mgpau.sandbox.nonprod-project-avengers.com.zip

cc Varun Velamuri