For tracking the issue observed with CBSE-7175.
- There are 7 FTS indexes and out of that only 3 are of scorch types and the remaining 4 are of older upside_down/moss index type.
- Even though, the bucket contains roughly about 2.5M documents, looks like there are mutations happening all the time?
- FTS services is provisioned with a meagre 2GB RAM for the ~7 partitions residing in every node.
- From a quick glance - each of the index definition has rich definition - meaning they certainly index 5-10 fields in most them.
- From the slow-queries logged, it seems that they always perform field scoped queries with no custom field sorting on results, still they have enabled `include_all` and `store` options for the field mappings in the index definitions. (un necessary)
- As Steve Yen already pointed out, conjunct query with 275 sub queries doesn't look to be the best/right way to perform search. This can result in slow queries due to the heavy DGM (2GB RAM) situation in this case.
- There are many 429 query reject errors happening due to "lack of resources/out of memory".
- `id` field indexed as a text as per the index definition.
Another wrinkle noted is the the number of files on disk grows with the scorch indexes. The customer might soon hit the a file handle threshold issue if this trend continues.
The number of files at root are always less than 25 here and it looks like it never touch 30 too for any of the scorch indexes. But still the total files on disk are mounting slowly and reached at this massive figure.
-Ongoing searches doesn't look like a problem at this point. Terms search start/end counts are matching here.
-Merger and persister are almost working together in all the indexes, but they are lagging a little behind the current root.
So this needs to be analysed further. Not yet have an explanation for this #file growth.