Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
5.1.2
-
Triaged
-
No
Description
Currently plasma mem tuner logic decrements memory quota and forces GC when the indexer process RSS overshoots user quota by 15% but the increment logic does not examine RSS before choosing to restore the quota.
This results in spiky RSS and frequent forced garbage collection.
Better to use examine RSS and only increment if the RSS is below user quota.
Inspiration: PID controller
Attachments
Issue Links
| For Gerrit Dashboard: MB-32433 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 103130,3 | MB-32433 mem: Only increment if RSS < quota | unstable | plasma | Status: MERGED | +2 | +1 |
| 103264,1 | MB-32433 Backport of mem: Only increment if RSS < quota | alice | plasma | Status: MERGED | +2 | +1 |
| 104529,1 | MB-32433 mem: Only increment if RSS < quota | vulcan | plasma | Status: ABANDONED | 0 | 0 |
Activity
Local testing shows RSS to be more stable, but Mihir Kamdar could you please help run any of your system tests to see if the RSS behavior can be verified?
thanks
Build couchbase-server-6.0.1-2020 contains plasma commit 53c7714 with commit message:
MB-32433 Backport of mem: Only increment if RSS < quota
Used the following test to validate the fix
1. 2 node cluster, kv+query on one node, index on another
2. Indexer memory quota = 4000 MB. Storage mode = Plasma
3. 1 bucket 1 index
4. Loaded 4M items of size ~1 KB.
5. Index resident ratio = 21%
Continuously monitored the memory_rss stat. Without the fix, I could see the spiky behavior because of which Adaptive memory quota tuning kicked in and incremented/decremented memory quota. In the ~20-30 mins timeframe of the test, there were 87 instances of incrementing memory quota, and 10 instances of decrementing it.
With the fix on 6.5.0-2021, there were only 2 instances in the 30 min timeframe of the test that memory quota was decremented, and it was never incremented. The RSS was always under the overall memory quota. The spiky behavior is gone.
Created MB-32544 to track the fix in Mad-hatter. Removing Mad-hatter from the FixVersion of this bug.
Re-opening the ticket to remove the 5.5.x candidate label.
See MB-32636 for the 5.5.4 release.
Build couchbase-server-6.5.0-2184 contains plasma commit b9bd18d with commit message:
MB-32433 mem: Only increment if RSS < quota
Need to examine average GC waste value -> gc_waste_avg (Available memory not released by go runtime and jemalloc)
Now we should lower plasma quota to a value less by gc_waste_avg
Take 5 samples of gc waste gc_waste_avg = RSS_before_GC_force - RSS_after_GC_waste
plasma_quota = 90% of indexer_quota - gc_waste_avg