[System Test]Rebalance out fts node taking more than 21 hrs

Build: 7.0.0-4502
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1

In the test,

• there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
• For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
• Continuously run queries on the indexes of collections of bucket1 and bucket2
• wait for 15 mins
• kill cbft on 172.23.97.217 and wait for 15 mins
• stop all mutations and wait for 10 mins
• add fts node 172.23.107.4 and start rebalance and wait for 15 mins
• stop create index loop on bucket4 and bucket5
• wait for rebalance to complete
• Once rebalance is complete, kill cbft on 172.23.107.5 and wait for 15 mins
• start mutations on the collections of bucket1, bucket2 and buckt3
• wait for 2 mins and rebalance to remove node 172.23.97.232 and wait for 5 mins
• kill memcached on 172.23.97.237 and wait for 15 mins
• Add data node 172.23.97.232 and rebalance and wait for 10 mins
• kill cbft on 172.23.107.5 ( which was at 2021-02-22T14:33:48-08:00) and wait for 15 mins
• Rebalance out fts node

Rebalance is running more than 21 hrs.

Note that, in the above test, there is no collection create/delete and no create/delete index loop.

Test step:

[2021-02-22T14:33:48-08:00, vijayviji/sshpass:f5b1df] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.107.5 kill -SIGKILL $(pgrep cbft)

[pull] sequoiatools/cmd

[2021-02-22T14:33:53-08:00, sequoiatools/cmd:c6c7ca] 900

[pull] sequoiatools/couchbase-cli:7.0

[2021-02-22T14:49:02-08:00, sequoiatools/couchbase-cli:7.0:df842f] rebalance -c 172.23.97.215:8091 --server-remove 172.23.97.217 -u Administrator -p password

Cluster config:
fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]
kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]
n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]

Logs:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.107.2.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.107.3.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.107.4.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.107.5.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.215.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.216.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.217.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.227.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.232.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.235.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.236.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/rebalance/collectinfo-2021-02-23T193554-ns_1%40172.23.97.237.zip