[System Test] Failover/rebalance stuck for more than 18 hrs

Build: 7.1.0-2030
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Scale: 1

Failover/rebalance stuck for more than 18 hrs:

[pull] sequoiatools/couchbase-cli:7.1

[2022-01-10T21:18:49-08:00, sequoiatools/couchbase-cli:7.1:de0fce] failover -c 172.23.106.253:8091 --server-failover 172.23.106.154:8091 -u Administrator -p password --hard

[pull] sequoiatools/couchbase-cli:7.1

[2022-01-10T21:19:00-08:00, sequoiatools/couchbase-cli:7.1:ccb433] rebalance -c 172.23.106.253:8091 -u Administrator -p password

Note that documents are loaded with expiry and eventually all docs are expired but seems some fts indexes not processed these mutations.

Logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.105.185.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.105.186.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.105.190.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.106.182.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.106.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.106.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.106.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.106.255.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.107.89.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.97.213.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1641882494/collectinfo-2022-01-11T062815-ns_1%40172.23.97.214.zip