FTS - Explore the feasibility of 1000 fts indexes in a single node.

Index definition:

{

"name": "$indexName",

"type": "fulltext-index",

"params": {

 "mapping": {

  "default_mapping": {

   "enabled": true,

   "dynamic": true

  },

  "default_type": "_default",

  "default_analyzer": "standard",

  "default_datetime_parser": "dateTimeOptional",

  "default_field": "_all",

  "store_dynamic": false,

  "index_dynamic": true,

  "docvalues_dynamic": false

 },

 "store": {

  "indexType": "scorch",

  "kvStoreName": ""

 },

 "doc_config": {

  "mode": "type_field",

  "type_field": "type",

  "docid_prefix_delim": "",

  "docid_regexp": ""

 }

},

"sourceType": "couchbase",

"sourceName": "travel-sample",

"sourceUUID": "a9da79e59360818ac7098e4dec6c2acf",

"sourceParams": {},

"planParams": {

 "maxPartitionsPerPIndex": 1024,

 "numReplicas": 0,

 "indexPartitions": 1

},

"uuid": ""

}

Details:

1. The experiment was performed on a single node cluster.
   #The search node had 8 cores, 31 GB RAM, 128 GB disk size ( on the mount point where couchbase server was installed).
2. The attached bash script, containing the REST endpoint for creating an index, has the ability to dispatch a user-defined number of index creation requests. The range of numbers between the bounds are suffixed to the index names to avoid conflicts while index creation. For example, the index names with lower bound = 1 and upper bound = 5 would create indexes travel_100K_1, travel_100K_2 … travel_100K_5. To run it, use 

   bash create_indexes.sh lower_bound upper_bound

1. Initially, a batch size (upper_bound - lower_bound + 1) of 10 index creation requests were dispatched and towards the end of the experiment (when there were lags in the UI) the batch size was reduced to 5.
2. The indexes were queried upon successful creation and behaved as expected. However there were hanging indexes (the number of docs processed were not complete) which gave partial results only for certain queries.
3. There were errors such as this: 

   collection_utils: refreshMetaFieldValCache, err: json: cannot unmarshal object into Go struct field BleveParams.mapping of type mapping.IndexMapping

   However, the bleve params were reflected in the planpindexes, as indicated in the attached logs.

The experiment was repeated multiple times, with two scenarios of 1M and 100K docs loaded in the bucket, and each time the disk usage by the fts service more or less equal to 12 GB, which was roughly 40% of the total space of its mounted disk partition and the number of zap+bolt files were roughly in the range of 850 - 900.

After increasing the bucket's RAM quota:

1. There are no hanging indexes. The number of indexes created is 580, the search service's RAM quota of 11 GB
   was exhausted. The data service is assigned 18 GB RAM. The disk usage was 101 GB.
2. While running the batches, the bucket's RAM quota had to be increased at times to reduce the latency of the index creation. Currently the bucket's RAM quota stands at around 9.8 GB.
3. Towards the end, the responsiveness of the UI drops, and the time taken to submit creation requests and complete the index building for a batch of 10 roughly varies between 10 to 15 mins.
4. Query latency on an average for a match query is 3 seconds.