FTS - Index creation hangs due to "Backfill ...pending errors"

FTS index creation stalls indefinitely on the latest CC builds when trying to create an index on custom scope1.collection1.

Steps involved.

1. Create a sample scope (scope1) and collection (collection1) under the beer-sample bucket. 

   curl -u Administrator:asdasd -i -X POST http://localhost:9000/pools/default/buckets/beer-sample/collections -d name=scope1

   curl -u Administrator:asdasd -X POST http://localhost:9000/pools/default/buckets/beer-sample/collections/scope1 -d name=collection1

1. Load documents into the scope1.collection1.

   ./cbimport json -c 127.0.0.1:9000 -b beer-sample -d file:///Users/sreekanth/Documents/beer-sample.json -f list -u Administrator -p asdasd --generate-key sample#MONO_INCR# --scope-collection-exp scope1.collection1

1.  Create an FTS index on the scope1.collection1.

   curl -XPUT -H "Content-Type: application/json" -u Administrator:asdasd http://localhost:9200/api/index/FTS -d \

   '{"type": "fulltext-index",

   "name": "FTS",

   "sourceType": "couchbase",

   "sourceName": "beer-sample",

   "planParams": {

   "maxPartitionsPerPIndex": 1024,

   "indexPartitions": 1

   },

   "params": {

   "doc_config": {

   "docid_prefix_delim": "",

   "docid_regexp": "",

   "mode": "scope.collection.type_field",

   "type_field": "type"

   },

   "mapping": {

   "analysis": {},

   "default_analyzer": "standard",

   "default_datetime_parser": "dateTimeOptional",

   "default_field": "_all",

   "default_mapping": {

   "dynamic": true,

   "enabled": false

   },

   "default_type": "_default",

   "docvalues_dynamic": true,

   "index_dynamic": true,

   "store_dynamic": false,

   "type_field": "_type",

   "types": {

   "scope1.collection1.brewery": {

   "dynamic": false,

   "enabled": true,

   "properties": {

   "description": {

   "dynamic": false,

   "enabled": true,

   "fields": [

   {

   "docvalues": true,

   "include_in_all": true,

   "include_term_vectors": true,

   "index": true,

   "name": "description",

   "type": "text"

   }

   ]

   }

   }

   }

   }

   },

   "store": {

   "indexType": "scorch"

   }

   },

   "sourceParams": {}

   }'

Wait for the index creation progress. It just stuck with the following errors in memcached logs.

2020-03-02T14:11:36.472953+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:637 is pending

2020-03-02T14:11:36.473013+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:601 is pending

2020-03-02T14:11:36.473036+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:811 is pending

2020-03-02T14:11:36.473061+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:154 is pending

2020-03-02T14:11:36.473082+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:731 is pending

2020-03-02T14:11:36.473100+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:38 is pending

2020-03-02T14:11:36.473136+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:27 is pending

2020-03-02T14:11:36.473164+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:957 is pending

2020-03-02T14:11:36.473239+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:741 is pending

2020-03-02T14:11:36.473266+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:536 is pending

2020-03-02T14:11:36.473286+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:2 is pending

2020-03-02T14:11:36.473304+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:639 is pending

2020-03-02T14:11:36.473328+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:369 is pending

2020-03-02T14:11:36.473372+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:245 is pending

2020-03-02T14:11:36.473395+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:334 is pending

2020-03-02T14:11:36.473417+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:979 is pending

2020-03-02T14:11:36.473435+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:887 is pending

2020-03-02T14:11:36.473459+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:940 is pending

2020-03-02T14:11:36.473494+05:30 INFO (beer-sample) Backfill for eq_dcpq:fts:FTS_217400cafff61466_4c1c5584-486a9053 vb:827 is pending