[System Test][ERRO]manager: could not open pindex path:err: cannot open index, metadata missing

Build: 7.0.0-4554

Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1

In the test,

• there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
• For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
• Continuously run queries on the indexes of collections of bucket1 and bucket2
• wait for 15 mins
• kill cbft on 172.23.97.217 and wait for 15 mins
• stop all mutations and wait for 10 mins
• add fts node 172.23.107.4 and start rebalance and wait for 15 mins
• stop create index loop on bucket4 and bucket5
• wait for rebalance to complete
• Once rebalance is complete, kill cbft on 172.23.107.5 and wait for 15 mins
• start mutations on the collections of bucket1, bucket2 and buckt3
• wait for 2 mins and rebalance to remove node 172.23.97.232 and wait for 5 mins
• kill memcached on 172.23.97.237 and wait for 15 mins
• Add data node 172.23.97.232 and rebalance and wait for 10 mins
• kill cbft on 172.23.107.5 ( which was at 2021-02-22T14:33:48-08:00) and wait for 15 mins
• Rebalance out fts node : 172.23.97.217 and wait for 15 mins
• Swap Rebalance (add 172.23.97.217: fts and remove 172.23.107.5:fts)
• Rebalance runs for long time more than 15 hrs and we see below error in the log:

172.23.97.217 : fts

/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-02-26T23:20:40.690-08:00 [ERRO] manager: could not open pindex path: /data/@fts/bucket_bucket2_idx_oyvwo-free_parking-type-country-public_likes_4674c43b06c48c5f_33f04ac1.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket2_idx_oyvwo-free_parking-type-country-public_likes_4674c43b06c48c5f_33f04ac1.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).LoadDataDir.func1() at manager.go:605 

This test is with :

• 20 indexes with 220 partitions overall.
• No mutations, no collections crud, no index create/drop loop running and do not see high resource usage (attached sceernshots)

Logs:

         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.107.2.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.107.3.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.107.4.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.107.5.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.215.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.216.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.217.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.227.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.232.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.235.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.236.zip
         url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614413844/collectinfo-2021-02-27T081725-ns_1%40172.23.97.237.zip