Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49262

Checkpoint expel stops before low mark

    XMLWordPrintable

Details

    Description

      Introduced in http://review.couchbase.org/c/kv_engine/+/163330

      Expelling checks after every vbucket if further reduction in memory usage is required

          const auto vbuckets = bucket.getVBuckets().getVBucketsSortedByChkMgrMem();
          for (const auto& it : vbuckets) {
              const auto vbid = it.first;
              VBucketPtr vb = bucket.getVBucket(vbid);
              if (!vb) {
                  continue;
              }
       
              const auto expelResult =
                      vb->checkpointManager->expelUnreferencedCheckpointItems();
              EP_LOG_DEBUG(
                      "Expelled {} unreferenced checkpoint items "
                      "from {} "
                      "and estimated to have recovered {} bytes.",
                      expelResult.count,
                      vbid,
                      expelResult.memory);
       
              if (bucket.getRequiredCheckpointMemoryReduction() == 0) {
                  // All done
                  return ReductionRequired::No;
              }
          }
      

      size_t KVBucket::getRequiredCheckpointMemoryReduction() const {
          const auto checkpointMemoryRatio = getCheckpointMemoryRatio();
          const auto checkpointQuota = stats.getMaxDataSize() * checkpointMemoryRatio;
          const auto recoveryThreshold =
                  checkpointQuota * getCheckpointMemoryRecoveryUpperMark();
          const auto usage = stats.getCheckpointManagerEstimatedMemUsage();
       
          if (usage < recoveryThreshold) {
              return 0;
          }
       
          const auto lowerRatio = getCheckpointMemoryRecoveryLowerMark();
          const auto lowerMark = checkpointQuota * lowerRatio;
          Expects(usage > lowerMark);
          const size_t amountOfMemoryToClear = usage - lowerMark;
       
          const auto toMB = [](size_t bytes) { return bytes / (1024 * 1024); };
          const auto upperRatio = getCheckpointMemoryRecoveryUpperMark();
          EP_LOG_DEBUG(
                  "Triggering memory recovery as checkpoint memory usage ({} MB) "
                  "exceeds the upper_mark ({}, "
                  "{} MB) - total checkpoint quota {}, {} MB . Attempting to free {} "
                  "MB of memory.",
                  toMB(usage),
                  upperRatio,
                  toMB(checkpointQuota * upperRatio),
                  checkpointMemoryRatio,
                  toMB(checkpointQuota),
                  toMB(amountOfMemoryToClear));
       
          return amountOfMemoryToClear;
      }
      

      getRequiredCheckpointMemoryReduction boils down to:

      If checkpoint memory usage exceeds high mark:
       -> amount of memory to recover to reach the low mark
      else:
       -> 0
      

      Checking after every vbucket means expelling will often stop slightly below the high mark.

      Anecdotally, this has been seen in cluster run to lead to each run of the ClosedUnrefCheckpointRemoverTask expelling from a single vbucket, then ending. This leads to a lot of logging of:

      ClosedUnrefCheckpointRemoverTask:0 Triggering checkpoint memory recovery - attempting to free X MB
      

      and a reduced rate of expelling (as the task needs to be retriggered/scheduled between each vbucket).

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1729 contains kv_engine commit d4915c5 with commit message:
            MB-49262: Ensure expelling continues until the low mark is reached

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1729 contains kv_engine commit d4915c5 with commit message: MB-49262 : Ensure expelling continues until the low mark is reached
            paolo.cocchi Paolo Cocchi added a comment -

            Oh I didn't spot that last line in the description, thanks for clarifying James. Anyway yes, my comment's intent was for remarking that this is important to fix, so we're on the same page

            paolo.cocchi Paolo Cocchi added a comment - Oh I didn't spot that last line in the description, thanks for clarifying James. Anyway yes, my comment's intent was for remarking that this is important to fix, so we're on the same page

            Indeed Paolo Cocchi, as noted in the description

            a reduced rate of expelling (as the task needs to be retriggered/scheduled between each vbucket)

            Whereas

            Under heavy load, checkpoint memory usage may stay above the high mark during expelling

            was in reference to the fact that under very high load, the memory usage may not drop below the high mark, so the task could continue visiting vbuckets in a single run in this situation. As a result, workloads like a restore from backup might not suffer as badly as may be expected - though they will likely still be impacted to some degree.

            It's definitely an issue in any case, but so far it appears to "only" be a perf issue; expelling should eventually recover memory, just much slower than it could.

            james.harrison James Harrison added a comment - Indeed Paolo Cocchi , as noted in the description a reduced rate of expelling (as the task needs to be retriggered/scheduled between each vbucket) Whereas Under heavy load, checkpoint memory usage may stay above the high mark during expelling was in reference to the fact that under very high load, the memory usage may not drop below the high mark, so the task could continue visiting vbuckets in a single run in this situation. As a result, workloads like a restore from backup might not suffer as badly as may be expected - though they will likely still be impacted to some degree. It's definitely an issue in any case, but so far it appears to "only" be a perf issue; expelling should eventually recover memory, just much slower than it could.
            paolo.cocchi Paolo Cocchi added a comment - - edited

            Under heavy load, checkpoint memory usage may stay above the high mark during expelling; in that case this issue would have little impact.

            It's true that this bug isn't preventing ItemExpel from running, but it seems that this is making ItemExpel much less effective than expected.
            Reason is that the expel Task will yield too early (ie, as soon as the CM mem-usage drops below the upper_mark). That means that a full logical ItemExpel run will split across multiple runs, which also means going through the scheduler every time.

            paolo.cocchi Paolo Cocchi added a comment - - edited Under heavy load, checkpoint memory usage may stay above the high mark during expelling; in that case this issue would have little impact. It's true that this bug isn't preventing ItemExpel from running, but it seems that this is making ItemExpel much less effective than expected. Reason is that the expel Task will yield too early (ie, as soon as the CM mem-usage drops below the upper_mark). That means that a full logical ItemExpel run will split across multiple runs, which also means going through the scheduler every time.
            drigby Dave Rigby added a comment -

            Marking as a regression - while the separate checkpoint quota is new in Neo, conceptually I believe we are no longer freeing memory that we previously used to pre-Neo.

            drigby Dave Rigby added a comment - Marking as a regression - while the separate checkpoint quota is new in Neo, conceptually I believe we are no longer freeing memory that we previously used to pre-Neo.
            drigby Dave Rigby added a comment -

            Note it appears this issue is showing up in recent perf tests (e.g. MB-49170) where checkpoint memory usage ends up "stuck" close to high watermark.

            drigby Dave Rigby added a comment - Note it appears this issue is showing up in recent perf tests (e.g. MB-49170 ) where checkpoint memory usage ends up "stuck" close to high watermark.

            Under heavy load, checkpoint memory usage may stay above the high mark during expelling; in that case this issue would have little impact.
            Under low load CM mem usage would hover near the high mark. In that case, the load may be low enough that expelling only to the high mark is still sufficient to avoid backpressure.
            This may be more problematic in buckets with very small quotas - there may be relatively little headroom beyond the high mark before the full CM quota is hit, leading to more frequent tmp fails than otherwise expected.

            Secondarily, this issue will likely lead to faster wrapping of memcached.log due to CM mem usage more frequently reaching the high mark, logging Triggering checkpoint memory recovery each time; this has been noted in local repros.

            james.harrison James Harrison added a comment - Under heavy load, checkpoint memory usage may stay above the high mark during expelling; in that case this issue would have little impact. Under low load CM mem usage would hover near the high mark. In that case, the load may be low enough that expelling only to the high mark is still sufficient to avoid backpressure. This may be more problematic in buckets with very small quotas - there may be relatively little headroom beyond the high mark before the full CM quota is hit, leading to more frequent tmp fails than otherwise expected. Secondarily, this issue will likely lead to faster wrapping of memcached.log due to CM mem usage more frequently reaching the high mark, logging Triggering checkpoint memory recovery each time; this has been noted in local repros.

            People

              james.harrison James Harrison
              james.harrison James Harrison
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty