Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32433

[plasma] MemTuner examine RSS before incrementing for smoother control

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.1.2
    • 6.0.1
    • storage-engine
    • Triaged
    • No

    Description

      Currently plasma mem tuner logic decrements memory quota and forces GC when the indexer process RSS overshoots user quota by 15% but the increment logic does not examine RSS before choosing to restore the quota.

      This results in spiky RSS and frequent forced garbage collection.

      Better to use examine RSS and only increment if the RSS is below user quota.

      Inspiration: PID controller 

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-32636 [BP 5.5.4] - [plasma] MemTuner examine RSS before incrementing for smoother control

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-32861 [Plasma-dgm] 15% slower indexing in DGM after MB-32433 change

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-32595 [FTS] reuse/incorporate latest GSI memory-mgmt improvements into FTS

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MB-32544 [5.1.3 BP MB-32433] - [plasma] MemTuner examine RSS before incrementing for smoother control

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-30373 Honour Index Service Memory Quota

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-32433
          # Subject Branch Project Status CR V
          103130,3 MB-32433 mem: Only increment if RSS < quota unstable plasma Status: MERGED +2 +1
          103264,1 MB-32433 Backport of mem: Only increment if RSS < quota alice plasma Status: MERGED +2 +1
          104529,1 MB-32433 mem: Only increment if RSS < quota vulcan plasma Status: ABANDONED 0 0

          Activity

            Ascending order - Click to sort in descending order
            sundar Sundar Sridharan (Inactive) added a comment -

            Need to examine average GC waste value -> gc_waste_avg (Available memory not released by go runtime and jemalloc)

            Now we should lower plasma quota to a value less by gc_waste_avg

            Take 5 samples of gc waste gc_waste_avg = RSS_before_GC_force - RSS_after_GC_waste

            plasma_quota = 90% of indexer_quota - gc_waste_avg

            sundar Sundar Sridharan (Inactive) added a comment - Need to examine average GC waste value -> gc_waste_avg (Available memory not released by go runtime and jemalloc) Now we should lower plasma quota to a value less by gc_waste_avg Take 5 samples of gc waste gc_waste_avg = RSS_before_GC_force - RSS_after_GC_waste plasma_quota = 90% of indexer_quota - gc_waste_avg
            sundar Sundar Sridharan (Inactive) added a comment -

            Local testing shows RSS to be more stable, but Mihir Kamdar could you please help run any of your system tests to see if the RSS behavior can be verified?

            thanks

            sundar Sundar Sridharan (Inactive) added a comment - Local testing shows RSS to be more stable, but Mihir Kamdar could you please help run any of your system tests to see if the RSS behavior can be verified? thanks
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.0.1-2020 contains plasma commit 53c7714 with commit message:
            MB-32433 Backport of mem: Only increment if RSS < quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.1-2020 contains plasma commit 53c7714 with commit message: MB-32433 Backport of mem: Only increment if RSS < quota
            mihir.kamdar Mihir Kamdar (Inactive) added a comment -

            Used the following test to validate the fix

            1. 2 node cluster, kv+query on one node, index on another
            2. Indexer memory quota = 4000 MB. Storage mode = Plasma
            3. 1 bucket 1 index
            4. Loaded 4M items of size ~1 KB.
            5. Index resident ratio = 21%

            Continuously monitored the memory_rss stat. Without the fix, I could see the spiky behavior because of which Adaptive memory quota tuning kicked in and incremented/decremented memory quota. In the ~20-30 mins timeframe of the test, there were 87 instances of incrementing memory quota, and 10 instances of decrementing it.

            With the fix on 6.5.0-2021, there were only 2 instances in the 30 min timeframe of the test that memory quota was decremented, and it was never incremented. The RSS was always under the overall memory quota. The spiky behavior is gone.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Used the following test to validate the fix 1. 2 node cluster, kv+query on one node, index on another 2. Indexer memory quota = 4000 MB. Storage mode = Plasma 3. 1 bucket 1 index 4. Loaded 4M items of size ~1 KB. 5. Index resident ratio = 21% Continuously monitored the memory_rss stat. Without the fix, I could see the spiky behavior because of which Adaptive memory quota tuning kicked in and incremented/decremented memory quota. In the ~20-30 mins timeframe of the test, there were 87 instances of incrementing memory quota, and 10 instances of decrementing it. With the fix on 6.5.0-2021, there were only 2 instances in the 30 min timeframe of the test that memory quota was decremented, and it was never incremented. The RSS was always under the overall memory quota. The spiky behavior is gone.
            mihir.kamdar Mihir Kamdar (Inactive) added a comment -

            Created MB-32544 to track the fix in Mad-hatter. Removing Mad-hatter from the FixVersion of this bug.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Created MB-32544 to track the fix in Mad-hatter. Removing Mad-hatter from the FixVersion of this bug.
            wayne Wayne Siu added a comment -

            Re-opening the ticket to remove the 5.5.x candidate label.
            See MB-32636 for the 5.5.4 release.

            wayne Wayne Siu added a comment - Re-opening the ticket to remove the 5.5.x candidate label. See MB-32636 for the 5.5.4 release.
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.5.0-2184 contains plasma commit b9bd18d with commit message:
            MB-32433 mem: Only increment if RSS < quota

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2184 contains plasma commit b9bd18d with commit message: MB-32433 mem: Only increment if RSS < quota

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              sundar Sundar Sridharan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty