Search 750 indexes: Rebalance to remove a node took 59 hrs to complete

Build : 7.1.0-1624
Test Script & Scope : -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

refer: https://docs.google.com/document/d/18MjY2VhGISeIFFQyaFWfXeso6KL8zausVsbyjNeZ1zQ/edit#heading=h.f1ombfvfxdfq

Rebalance to remove node 172.31.40.108 took ~59 hrs. Its seems like a long time and not a good experience for user/administrator. With 500 indexes, it took ~ 10 hr 30 mins for the same operation. And with 750 indexes the time for this operation increased ~6X.

Test step:

[2021-11-03T17:18:31Z, sequoiatools/couchbase-cli:7.1:05429a] rebalance -c 172.31.39.188:8091 --server-remove 172.31.40.108 -u Administrator -p password

[2021-11-06T04:09:32Z, sequoiatools/cmd:859f30] 60

Logs collected at 2021-11-06 05:07:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636174270/collectinfo-2021-11-06T045111-ns_1%40172.31.39.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636174270/collectinfo-2021-11-06T045111-ns_1%40172.31.40.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636174270/collectinfo-2021-11-06T045111-ns_1%40172.31.43.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636174270/collectinfo-2021-11-06T045111-ns_1%40172.31.43.62.zip

Note that rebalance to remove 172.31.40.108 completed at 2021-11-06T04:09:32Z and test added the same node to the cluster at 2021-11-06T04:30:59Z, so we have all 4 nodes logs in the cbcollect logs.