Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27663

Improve backpressure/tmpOOM threshold when DWQ is large

    XMLWordPrintable

Details

    Description

      As observed in CBSE-4539, ep-engine has not sent temporary backoffs (tempOOM) back to clients when the DWQ exceeds a fixed number of items (1M) since 3.0.1 - see http://review.couchbase.org/#/c/46614/.

      Instead, it only sends tempOOM when the mem_used of the bucket exceeds a given percentage of the bucket quota - default 99%:

              "replication_throttle_threshold": {
      		 
                  "default": "99",

      Given that you'll only normally reach 99% usage after ~all items have been evicted and all the memory is being used for the checkpoints (in a full-eviction bucket), this isn't a very effective limit - it kicks in far too late.

      We should investigate an alternative mechanism. Possible ideas:

      • Trigger when checkpoint memory usage (CheckpointManager::getMemoryUsage) exceeds a percentage of the bucket quota.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-38855 [Magma] Memcached is OOM killed by kernel.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-39300 Cap checkpoint memory usage to % of quota

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-40644 Magma S0 tests on 7.0 hit out of memory issue

          • Major - Major loss of function.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-46827 Limit the Checkpoint memory usage

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-26494 Cursor Dropping - Improve Trigger Policy

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-29227 Couchbase Server can get into a livelock state due to high checkpoint memory usage

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              drigby Dave Rigby (Inactive)
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty