[System Test] : Some FTS queries erroring out with "context deadline exceeded"

Build : 7.0.0-4122
Test : -test tests/fts/cheshire-cat/test_fts_steady_state_cheshire_cat_basic.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Scale : 2
Day : 1

This FTS test does the following :
1. Cluster of 11 nodes = 6 KV + 5 FTS
2. 3 buckets with total of 20 scopes and 200 collections across all 3 buckets.
3. 2 more buckets with no non-default scopes and collections
4. Constant KV load on all collections of bucket 1,2,3. KV load is a mix of inserts, updates, deletes. Max number of items per collection = 100000, with all items having TTL set to 4 hrs. Size of doc = ~1000 bytes. Sample doc attached. catapult_sample_doc.json
5. Create 20 FTS indexes across all 3 buckets. Some indexes are multi-collection indexes, and some indexes are single collection indexes.
6. Running FTS queries across these indexes in a loop for 2 hrs. Queries are run via a multithreaded (5 worker threads for now) application. Each worker thread would pick up a query from the pool once it is done executing the previous one.
7. Drop all FTS indexes
8. Repeat from Step 4.

Checking the logs on one of the FTS nodes - 172.23.105.109, there are some queries that have errors.

2021-01-04T05:22:52.459-08:00 [ERRO] grpc_client: recv err: rpc error: code = DeadlineExceeded desc = context deadline exceeded -- cbft.(*GrpcClient).SearchRPC() at grpc_client.go:274

2021-01-04T05:22:52.459-08:00 [WARN] grpc_client: Query() returned error from host: 172.23.105.112:9130, err: grpc_client: query got status code: 504, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc114bc1ac0), Request:(*bleve.SearchRequest)(0xc0902b0600), Hits:search.DocumentMatchCollection(nil), Total:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = DeadlineExceeded desc = context deadline exceeded -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:161

2021-01-04T05:22:52.459-08:00 [WARN] grpc_client: scatter-gather error while awaiting results from host: 172.23.105.112:9130, err: context deadline exceeded -- cbft.(*GrpcClient).SearchInContext() at grpc_client.go:172

There are more such query failures as well later in the test. These should be investigated -
fts.log:2021-01-04T09:00:35.602-08:00 [ERRO] grpc_client: recv err: rpc error: code = DeadlineExceeded desc = context deadline exceeded – cbft.(*GrpcClient).SearchRPC() at grpc_client.go:274
fts.log:2021-01-04T09:02:11.068-08:00 [ERRO] grpc_client: recv err: rpc error: code = DeadlineExceeded desc = context deadline exceeded – cbft.(*GrpcClient).SearchRPC() at grpc_client.go:274
fts.log:2021-01-04T09:02:46.693-08:00 [ERRO] grpc_client: recv err: rpc error: code = DeadlineExceeded desc = context deadline exceeded – cbft.(*GrpcClient).SearchRPC() at grpc_client.go:274
fts.log:2021-01-04T09:03:17.369-08:00 [ERRO] grpc_client: recv err: rpc error: code = DeadlineExceeded desc = context deadline exceeded – cbft.(*GrpcClient).SearchRPC() at grpc_client.go:274