Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46827

Limit the Checkpoint memory usage

    XMLWordPrintable

Details

    Description

      Currently checkpoints can take up to the entire bucket quota. That has a number of problematic side effects in DGM scenarios, eg Resident Ratio dropping to 0, high replica checkpoints memory usage contributing to replication live-locks, etc..

      It's important to highlight that capping the Checkpoint quota is expected to directly solve some existing issues (like the Resident Ratio one), but the same doesn't apply to other issues (like the DCP live-lock) where the high Checkpoint mem-usage is probably just one of the factors that contribute to the problem. Other factors are being investigated and addressed in dedicated MBs.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-39344 [checkpoint] Allocation in replica checkpoints can push the system to hard OOM

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-27663 Improve backpressure/tmpOOM threshold when DWQ is large

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-46827
          # Subject Branch Project Status CR V
          155542,6 MB-46827: Introduce the 'checkpoint_memory_ratio' configuration param master kv_engine Status: MERGED +2 +1
          155543,6 MB-46827: Store reference to KVBucket in VBucket master kv_engine Status: MERGED +2 +1
          155544,12 MB-46827: Introduce KVBucket::hasCapacityInCheckpoints() master kv_engine Status: MERGED +2 +1
          156006,9 MB-46827: Remove replication write-queue caps master kv_engine Status: MERGED +2 +1
          156101,3 MB-46827: Fix build master kv_engine Status: MERGED +2 +1
          157352,7 MB-46827: Track bucket checkpoint mem-usage in EPStats::coreLocal master kv_engine Status: MERGED +2 +1
          157353,5 MB-46827: Cache checkpoint_memory_ratio config param in KVBucket master kv_engine Status: MERGED +2 +1
          157399,4 MB-46827: Make the checkpoint_memory_ratio config param dynamic master kv_engine Status: MERGED +2 +1
          157520,6 MB-46827: Remove cursor_dropping_<lower/upper>_mark master kv_engine Status: MERGED +2 +1
          157522,5 MB-46827: Remove VBucketMap::getVBucketsTotalCheckpointMemoryUsage master kv_engine Status: MERGED +2 +1
          157579,4 MB-46827: Trigger checkpoint memory recovery regardless of LWM master kv_engine Status: MERGED +2 +1
          157591,4 MB-46827: Introduce the 'checkpoint_memory_recovery_upper_mark' param master kv_engine Status: MERGED +2 +1
          157603,16 MB-46827: Introduce the new checkpoint memory recovery logic master kv_engine Status: MERGED +2 +1
          157659,11 MB-46827: Remove cursor_dropping_checkpoint_mem_<lower/upper>_mark master kv_engine Status: MERGED +2 +1
          157660,3 MB-46827: Introduce the 'checkpoint_memory_recovery_lower_mark' param master kv_engine Status: MERGED +2 +1
          157730,12 MB-47386: Remove 'keep_closed_chks' config param master kv_engine Status: MERGED +2 +1

          Activity

            People

              paolo.cocchi Paolo Cocchi
              paolo.cocchi Paolo Cocchi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty