[System Test] Rebalancing FTS node takes infinite amount of time

While rebalancing the fts node, it takes infinite amount of time. The rebalance is stuck at 18.2% for more than one hour. Can be seen in the image attached.

Steps:

1. Created an on-prem cluster with 12 nodes. Out of which 5 are fts nodes.
2. Each node has atleast 12gb of memory.
3. Created one scope and one collection under that scope
4. Loaded 3 million documents using the sift dataset
5. Created one index with 90 partitions acoss 5 fts nodes.
6. Run knn queries
7. Mutated the documents and then again ran the knn query
8. Rebalanced in an FTS node
9. Perform mutations and then again run the knn queries
10. Rebalanced out an FTS node

The above steps were performed in loop. After 6 hours or so, the FTS node rebalance in was stuck at 18.2% and it is taking infinite time to perform rebalance.

Attached are the logs of the cluster:
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.105.122.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.106.171.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.106.176.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.106.30.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.96.198.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.96.230.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.96.245.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.97.100.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.97.108.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.97.66.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1714674198/collectinfo-2024-05-02T182621-ns_1%40172.23.97.67.zip