FTS rebalance operation failed with agent_died, noconnection error

Test Steps

1. Deploy a capella AWS cluster having following configuration -

   Service	Number of nodes	CPU cores	Memory	Disk size
   KV	3	8	32GB	250GB
   FTS	5	16	32GB	250B
   N1QL	2	4	16GB	250GB

2. Create a bucket named sift_bucket and sift_scope0.sift_collection0 keyspace within it.
3. Load 20m documents containing vector data having 1536 dimensions within the collection created in previous step.
4. Create a vector index on vector field and wait for the index building process to complete.
5. Start knn query workload in the background.
6. Rebalance in 1 FTS node.

Observation

Rebalance operation failed with agent_died, noconnection error.

2024-02-16T11:05:30.746Z, ns_orchestrator:0:critical:message(ns_1@svc-q-node-009.eow7pjgyi8md4tyn.sandbox.nonprod-project-avengers.com) - Rebalance exited with reason {service_rebalance_failed,fts,

                                 {agent_died,<36734.3317.0>,noconnection}}.

Rebalance Operation Id = 8cc8093243c730b95fda268119ccca7b

Rebalance operation succeeded on second attempt, hence marking this ticket as major rather than critical.