Search service rebalance hung

Version: couchbase-cloud-server-7.2.1-5819-v1.0.20

Test Scenario : Test code

Steps involved in the test:

1. _{15 scopes and 60 collections across 3 buckets}
   • _{Each bucket -> 10M data 1 index on each collection -> 60 indexes}
2. _{Sleep - 15 mins}
3. _{Run FTS Queries FTS Flex Queries Random fashion for 2 hrs}
4. _{Kill CBFT}
5. _{sleep for 15 mins}
6. _{Scale out to 5 nodes}
7. _{Create 10 more fts indexes with following config :}
   • _{1 Index | 1 Replica | 5 Partitions}
   •  _{3 Indexes | 2 Replicas | 6 Partitions}
   • _{6 Indexes | 0 Replicas | 4 Partitions}
8. _{Sleep for 15 mins}
9. _{Run FTS Queries FTS Flex Queries Random fashion for 30 mins again}
10.  _{Kill CBFT}
11. _{Sleep for 15 mins}
12. _{Rebalance/Scale in to 4 nodes}

During this we see rebalance for search service hung and to be running for more than 25Hrs

FYI : The disk and CPU utilisation for all nodes seem to be in control and healthy.