[System Test]:Rebalance to remove kv node taking more than 22 hrs

Build: 7.0.0-4476
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1

Rebalance to remove kv node taking more than 22hrs at search rebalance and still running.

Test Step:

[2021-02-18T11:01:19-08:00, sequoiatools/cmd:5360a7] 60

[pull] sequoiatools/cmd

[2021-02-18T11:02:27-08:00, sequoiatools/cmd:a55189] 900

[pull] vijayviji/sshpass

[2021-02-18T11:17:46-08:00, vijayviji/sshpass:886f18] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.107.5 kill -SIGKILL $(pgrep cbft)

[pull] sequoiatools/cmd

[2021-02-18T11:17:51-08:00, sequoiatools/cmd:fcca54] 900

[pull] sequoiatools/couchbase-cli:7.0

[2021-02-18T11:32:59-08:00, sequoiatools/couchbase-cli:7.0:e5b3a7] rebalance -c 172.23.97.215:8091 --server-remove 172.23.97.232 -u Administrator -p password

In the test,

• there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
• For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.

Cluster config:
fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]
kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]
n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]

Logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613758874/collectinfo-2021-02-19T182119-ns_1%40172.23.97.237.zip