[System Test] Index rebalance hung due to outstanding GetCollectionSeqnos

Build : 7.0.0-4678
Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_dgm.yml
Scale : 2
Iteration : 4th

This is a variant of the GSI component test where we are trying to push indexes to have low Resident ratio (upto 20%). We are not really upto 20%, but somewhere between 60-70% overall.

In the 4th iteration, we see a rebalance operation that started around 2021-03-14T18:40:10 being stuck for about 15 hrs now. All the indexes are online, none of them are in moving state. Attached is the latest output from getIndexStatus. This does not look to be like MB-44845 or MB-44506 where there were indexes in Moving state causing index rebalance to be stuck.

Index nodes : 172.23.105.186, 172.23.105.190, 172.23.106.154, 172.23.106.255, 172.23.97.213, 172.23.97.214

Logs have been uploaded to supportal : https://supportal.couchbase.com/snapshot/64c10cbfcaee4fd7b4d7d21ebef768dd::0

Marking this as a Test Blocker as the test cannot proceed without manual intervention (to abort the rebalance)