Create index query requested terminated after 38m even though index created successfully

Build: 7.0.0 build 4122

Test - disk_full scenario:

• Have a 4 node cluster with index on all the nodes
• Create 2 scopes and 5 collections in each scope
• continuously load data to all collections
• In parallel, run create secondary indexes on all collections, drop indexes and scan indexes
• In parallel, sequentially fill the disk and recover after 60 secs on each index nodes. Wait for 5 mins till service recovers.

In this case, one of the create index request terminated after 38m, even though index creation is successfully.

1. Create index request initiated at 2021-01-12 19:40:24 on query node 172.23.121.49

   2021-01-12 19:40:24 | INFO | MainProcess | index_create_task_manager | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX idx_test_scope_5_test_collection_3job_title34 ON default:test_bucket.test_scope_5.test_collection_3(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': False}

   2021-01-12 19:40:24 | INFO | MainProcess | index_create_task_manager | [rest_client.query_tool] query params : statement=CREATE+INDEX+idx_test_scope_5_test_collection_3job_title34+ON+default%3Atest_bucket.test_scope_5.test_collection_3%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+False%7D

2. 172.23.121.47 received create index request at 2021-01-12T19:40:25.46
3. Test filled up the disk in 172.23.121.47 at 19:40:27.
4. Test cleared the disk in 172.23.121.47 at 19:41:28
5. Index idx_test_scope_5_test_collection_3job_title34 became active at :2021-01-12T19:40:29.593

   2021-01-12T19:40:29.593-08:00 [Info] ClustMgr::handleIndexMap, updated instances:

   InstanceId: 797366104283663239 Name: idx_test_scope_5_test_collection_3job_title34 Keyspace: test_bucket:test_scope_5:test_collection_3 State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0

6. query terminated at 2021-01-12T20:19:02.489

   2021-01-12T20:19:02.489-08:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.121.47:9100. Error = read tcp 172.23.121.49:32832->172.23.121.47:9100: use of closed network connection. Kill Pipe.

   2021-01-12T20:19:02.489-08:00 [Info] CreateIndex 0 test_bucket test_scope_5 test_collection_3/idx_test_scope_5_test_collection_3job_title34 using:GSI exprType:N1QL whereExpr:<ud>((`job_title` is not null))</ud> secExprs:<ud>([`job_title`])</ud> desc:[false] isPrimary:false scheme:SINGLE partitionKeys:<ud>([])</ud> with:{"defer_build":false} - elapsed(38m37.617110203s) err(Terminate Request due to client termination

It took 38m37.617110203s to terminate the request, even though index created successfully.

Logs:

https://cb-jira.s3.us-east-2.amazonaws.com/logs/query-timeout/collectinfo-2021-01-13T195550-ns_1%40172.23.121.45.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/query-timeout/collectinfo-2021-01-13T195550-ns_1%40172.23.121.47.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/query-timeout/collectinfo-2021-01-13T195550-ns_1%40172.23.121.49.zip
https://cb-jira.s3.us-east-2.amazonaws.com/logs/query-timeout/collectinfo-2021-01-13T195550-ns_1%40172.23.121.52.zip