Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
centos 7
-
Untriaged
-
No
-
FTS Sprint Jul-28-2018, FTS Sprint Aug-11-2018 (Beta)
Description
My setup:
node211: data service, 256GB RAM
node39: FTS service, 10GB RAM, FTS quota is 1GB
node40: FTS service, 10GB RAM, FTS quota is 1GB
10M documents, wiki dataset
FTS index size is 14GB, so 7GB per node
Test:
"wild card" queries and docs mutations (mutation rate is 1K docs/sec)
Behavior:
Phase 1 - indexing.
While indexing memory consumption by cbft is up to 3.5GB
Index is built w/o any issues
Phase 2 - warm-up
During warm-up phase client runs "wild card" queries only (no kv mutations)
The memory consumptions remains on 3.5-3.6 GB.
Consistent query throughput, no issues
Phase 3 - perf test
Once the amount of items in "Items to persist" starts to grow the cbft memory consumption quickly goes up. With just about 500 items in the queue cbft allocates another 2GB of RAM
With 1K+ items it takes all available memory (in my setup) and node gets failed-over due to OOM issues
Validating other queries and memory setups...
Logs from one of the runs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-enduser-469/172.23.99.211.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-enduser-469/172.23.99.39.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-enduser-469/172.23.99.40.zip
Attachments
Issue Links
- relates to
-
MB-28847 [FTS] Memory usage during index build overshoots RAM Quota
- Closed