Search: 1K indexes: One of index creation failed with i/o timeout

Build: 7.1.0-1566
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

• Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
• Create 1 bucket, 100 scopes and 10 collections in each scopes
• Create 2500 GSI indexes ( 5 on each collection)
• Load documents on some of the collections
• Created 1000 indexes: one index (1 partition) on each collection
• Run queries on each collection
• Mutate the documents on each collection and wait for all the index to process mutation
• Run queries on each collection
• Delete all the indexes

During creation of 1000 indexes, one of the index which is requested to create on scope_78.coll_1, failed with below error:

test log:

{"log":"2021-10-28 06:16:25,854 - ftsindexmanager - INFO - Creating FTS index bucket_bucket1_idx_mmxqj_8-country on ['scope_78.coll_1'] with 0 replicas and 1 partitions\n","stream":"stderr","time":"2021-10-28T13:16:25.854872741Z"}

{"log":"2021-10-28 06:16:48,697 - ftsindexmanager - INFO - Content = b'{\"error\":\"rest_auth: preparePerms, err: read tcp4 172.23.100.161:8094-\\\\u003e172.23.107.77:41203: i/o timeout\",\"request\":\"\",\"status\":\"fail\"}\\n' \n","stream":"stderr","time":"2021-10-28T13:16:48.697712238Z"}

{"log":"Response = {'content-type': 'application/json', 'x-content-type-options': 'nosniff', 'date': 'Thu, 28 Oct 2021 13:16:48 GMT', 'content-length': '140', 'connection': 'close', 'status': '400'}\n","stream":"stderr","time":"2021-10-28T13:16:48.697762553Z"}

Do not see a significant info in fts log for this request. But here are the logs at the above timestamp.

Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635430507/collectinfo-2021-10-28T141509-ns_1%40172.23.100.161.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635430507/collectinfo-2021-10-28T141509-ns_1%40172.23.100.162.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635430507/collectinfo-2021-10-28T141509-ns_1%40172.23.100.163.zip