Index rebuild did not complete after updating index during failover of (kv,fts) node

Build: 7.0.0-5127
Test job: centos-fts_moving-topology-scorch_5.5_P1
Test: ./testrunner -i /tmp/testexec.69218.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P1,index_type=scorch,fts_quota=990,get-cbcollect-info=True -t fts.moving_topology_fts.MovingTopFTS.update_index_during_failover,items=100000,cluster=D+F+F,GROUP=P1,index_partitions=20,get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,index_type=scorch,fts_quota=990

Test steps:

• Create a 3 node cluster with kv, kv+fts, kv+fts,
• Load 100k documents to bucket
• Create fts index with 20 partitions:
• When the above index create reaches 50%, update the index and parallely failover node n2(kv,fts)(172.23.100.17)

  {

  "type": "fulltext-index",

  "name": "default_index_1",

  "uuid": "64f53eb0745539c2",

  "params": {

  "store": {

  "kvStoreName": "mossStore",

  "mossStoreOptions": {},

  "indexType": "scorch"

  }

  },

  "sourceType": "couchbase",

  "sourceName": "default",

  "sourceUUID": "",

  "planParams": {

  "numReplicas": 0,

  "maxPartitionsPerPIndex": 64,

  "indexPartitions": 20

  },

  "sourceParams": {}

  }

• We see index build is stuck at 49205 docs as seen below after update:

[2021-05-13 11:35:36,733] - [rest_client:2449] INFO - Node 172.23.100.17 not part of cluster inactiveFailed

[2021-05-13 11:35:36,741] - [rest_client:2363] INFO - http://172.23.100.18:8091/pools/default/buckets/default/stats?zoom=minute

[2021-05-13 11:35:37,104] - [fts_base:4496] INFO - Docs in bucket = 66587, docs in FTS index 'default_index_1': 9917

[2021-05-13 11:35:43,119] - [rest_client:2449] INFO - Node 172.23.100.17 not part of cluster inactiveFailed

[2021-05-13 11:35:43,128] - [rest_client:2363] INFO - http://172.23.100.18:8091/pools/default/buckets/default/stats?zoom=minute

[2021-05-13 11:35:43,477] - [fts_base:4496] INFO - Docs in bucket = 100000, docs in FTS index 'default_index_1': 49205

[2021-05-13 11:35:49,493] - [rest_client:2449] INFO - Node 172.23.100.17 not part of cluster inactiveFailed

[2021-05-13 11:35:49,502] - [rest_client:2363] INFO - http://172.23.100.18:8091/pools/default/buckets/default/stats?zoom=minute

[2021-05-13 11:35:49,847] - [fts_base:4496] INFO - Docs in bucket = 100000, docs in FTS index 'default_index_1': 49205

[2021-05-13 11:35:55,862] - [rest_client:2449] INFO - Node 172.23.100.17 not part of cluster inactiveFailed

[2021-05-13 11:35:55,871] - [rest_client:2363] INFO - http://172.23.100.18:8091/pools/default/buckets/default/stats?zoom=minute

[2021-05-13 11:35:56,212] - [fts_base:4496] INFO - Docs in bucket = 100000, docs in FTS index 'default_index_1': 49205

[2021-05-13 11:36:02,227] - [rest_client:2449] INFO - Node 172.23.100.17 not part of cluster inactiveFailed

[2021-05-13 11:36:02,235] - [rest_client:2363] INFO - http://172.23.100.18:8091/pools/default/buckets/default/stats?zoom=minute

[2021-05-13 11:36:02,574] - [fts_base:4496] INFO - Docs in bucket = 100000, docs in FTS index 'default_index_1': 49205

Logs:
test_2 (1).zip
Console test log: consoleText