[System Test] service_rebalance_failed,fts, : rebalance: unknown nodes in nodesToRemoveParam

Build: 7.0.0-5127
Componet Clusterops test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1

Test WITH index create/drop loop. Note that rebalances in this test have hung because of MB-43690. So manually stopped so the test proceeds further.

In the test,

• there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
• For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
• Continuously run queries on the indexes of collections of bucket1 and bucket2
• wait for 15 mins
• kill cbft on 172.23.97.217 and wait for 15 mins
• stop all mutations and wait for 10 mins
• add fts node 172.23.107.4 and start rebalance and wait for 15 mins - (hanged - stopped rebalance manually)
• wait for rebalance to complete
• Once rebalance is complete, kill cbft on 172.23.107.5 and wait for 15 mins
• start mutations on the collections of bucket1, bucket2 and buckt3
• wait for 2 mins and rebalance to remove node 172.23.97.232 and wait for 5 mins (hanged - stopped rebalance manually)
• kill memcached on 172.23.97.237 and wait for 15 mins
• Kill cbft on 172.23.107.5 and wait for 15 mins
• rebalance out fts node : 172.23.97.217 (hanged - stopped rebalance manually)
• rebalance out same fts node : 172.23.97.217, which failed with below

2021-05-12T15:17:10.994-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.215) - Rebalance exited with reason {service_rebalance_failed,fts,

                              {worker_died,

                               {'EXIT',<0.27937.452>,

                                {rebalance_failed,

                                 {service_error,

                                  <<"rebalance: unknown nodes in nodesToRemoveParam: []string{\"94b3b49047824dadc31078c8176950dc\"}">>}}}}}.

Rebalance Operation Id = 1445227c58b9655f4c3632ba020573aa

Logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1620860270/collectinfo-2021-05-12T225751-ns_1%40172.23.97.237.zip

Cluster spec:

########## Cluster config ##################

######  fts : 6 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.4:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]  ###########

######  kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]  ###########

######  n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]  ###########