[System Test] Rebalance exited for FTS Node

Rebalance failed for the cluster with following error.

Rebalance exited with reason {service_rebalance_failed,fts,

{agent_died,<34981.3228.183>,

{lost_connection,

{'ns_1@172.23.106.30',shutdown}}}}.

Rebalance Operation Id = 7ae982048365c414e51ffd073833cb10 

Please refer to the logs for more details.

Steps:

1. Created an on-prem cluster with 12 nodes. Out of which 5 are fts nodes.
2. Loaded 100million documents. Each node has atleast 12gb of memory.
3. Created one scope and four collections under that scope
   • coll_0: Normal Vector Data (5m documents)
   • coll_1: Vector data with xattr (1m documents)
   • coll_2: Vector data with base_64 (1m documents)
   • coll_3: Vector data (1m documents)
4. Creating four indexes with 12 partitions across 5 fts nodes.
   • index_0: Indexing Normal Vector Data (5m documents)
   • index_1: Indexing Vector data with xattr (1m documents)
   • index_2: Indexing Vector data with base_64 (1m documents)
   • index_3: Indexing Vector data (1m documents)
5. Run 50 parallel knn queries on all the indexes created
6. Mutated the documents and then again run the 50 parallel knn query on all the indexes
7. Add an FTS node
8. Perform mutations and then again run the 50 parallel knn queries on all the indexes
9. Remove an FTS node

The above steps are performed in a loop.

Logs:

• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.106.176.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.106.30.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.96.198.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.96.230.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.96.245.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.97.100.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.97.66.zip
• https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.97.67.zip

Could get the logs only of the above nodes.