Need to examine average GC waste value -> gc_waste_avg (Available memory not released by go runtime and jemalloc)

Now we should lower plasma quota to a value less by gc_waste_avg

Take 5 samples of gc waste gc_waste_avg = RSS_before_GC_force - RSS_after_GC_waste

plasma_quota = 90% of indexer_quota - gc_waste_avg

Local testing shows RSS to be more stable, but Mihir Kamdar could you please help run any of your system tests to see if the RSS behavior can be verified?

thanks

Build couchbase-server-6.0.1-2020 contains plasma commit 53c7714 with commit message:
MB-32433 Backport of mem: Only increment if RSS < quota

Used the following test to validate the fix

1. 2 node cluster, kv+query on one node, index on another
2. Indexer memory quota = 4000 MB. Storage mode = Plasma
3. 1 bucket 1 index
4. Loaded 4M items of size ~1 KB.
5. Index resident ratio = 21%

Continuously monitored the memory_rss stat. Without the fix, I could see the spiky behavior because of which Adaptive memory quota tuning kicked in and incremented/decremented memory quota. In the ~20-30 mins timeframe of the test, there were 87 instances of incrementing memory quota, and 10 instances of decrementing it.

With the fix on 6.5.0-2021, there were only 2 instances in the 30 min timeframe of the test that memory quota was decremented, and it was never incremented. The RSS was always under the overall memory quota. The spiky behavior is gone.

Created MB-32544 to track the fix in Mad-hatter. Removing Mad-hatter from the FixVersion of this bug.

Re-opening the ticket to remove the 5.5.x candidate label.
See MB-32636 for the 5.5.4 release.

Build couchbase-server-6.5.0-2184 contains plasma commit b9bd18d with commit message:
MB-32433 mem: Only increment if RSS < quota