Details
-
Improvement
-
Resolution: Fixed
-
Major
-
7.2.0, 7.1.3
Description
Indexer data pipeline overheads can be reduced for low configuration nodes. E.g.
Consider a Capella configuration of 3 nodes with 16 Gig ram each, 48G total. With data + index co-located, the quota available is 80% i.e. 12.8GB. Index service gets 50% = 6.3GB. If data+index+query is co-located, index quota = 4.2GB.
With such configuration, it is better to reduce allocations throughout the pipeline which are generally done to achieve throughput of the order of 500k ops/sec.
Configs to consider:
"indexer.mutation_manager.maxQueueMem"
"indexer.settings.largeSnapshotThreshold"
"indexer.settings.minVbQueueLength"
"indexer.settings.sliceBufSize"
"indexer.numSliceWriters"
"projector.vbucketWorkers"
"projector.dcp.numConnections"
"projector.dcp.dataChanSize"
"indexer.settings.build.batch_size" - By default 5, reduce it for low config nodes
Use different configuation for slice during init vs incr build as incr build throughput is generally much lower than initial build:
"indexer.settings.sliceBufSize"
"indexer.numSliceWriters"
This can either be done dynamically or Capella control plane can push the configuration based on pre-defined rules as all the settings are already configurable.