[System Test] Rebalance stuck while upgrading indexer node

In the system test while upgrading(via swap rebalance) one of the index nodes from 7.2.3-6017 to 7.6.0-2024 the rebalance was stuck for 20+ hrs.

Rebalance progress - 30%

Cluster configuration initially was kv-5, index-5, query-2 The KV nodes were upgraded, then one query node was upgraded and two index nodes were upgraded via swap rebalance ( removing the old node and adding the upgraded node and triggering a rebalance). While upgrading the third index node the rebalance is stuck. Note - All upgraded nodes are installed in provisioned mode 

Steps followed for upgrading for the third index node - 

• Add the upgraded node(7.6 installed in provisioned mode) to the cluster
• Failover the node to be rebalanced out 
• Trigger the rebalance

Iteration 101 

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705990572/collectinfo-2024-01-23T062515-ns_1%40172.23.97.67.zip

iteration 96
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705969881/collectinfo-2024-01-23T004018-ns_1%40172.23.97.67.zip

iteration 91
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.106.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1705949228/collectinfo-2024-01-22T185601-ns_1%40172.23.97.67.zip