Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49983

CheckpointDestroyerTask unable to keep up with heavy write workloads

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      During heavy write loads, the CheckpointDestroyerTask has been observed to be taking many seconds to execute:

      2021-12-08T09:59:59.896711+00:00 WARNING (No Engine) Slow runtime for 'Destroying closed unreferenced checkpoints' on thread NonIoPool0: 6162 ms
      

      This results in a large number of Checkpoints being outstanding for deletion, along with large amounts of memory being tied up in them (pending_destruction_bytes):

      This causes much lower resident ratio than we would otherwise have, and increases tempOOM backoffs sent to clients.

      It also results in the NonIO thread being tied up and otherwise unable to run other NonIO tasks.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.1.0-1878 contains platform commit c34084f with commit message:
          MB-49983: MemoryTrackingAllocator: support sized deallocation

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1878 contains platform commit c34084f with commit message: MB-49983 : MemoryTrackingAllocator: support sized deallocation
          drigby Dave Rigby added a comment -

          Re-running same workload with https://review.couchbase.org/c/platform/+/167243, pending_destruction_bytes is significantly lower - a few 10s of MB:

          drigby Dave Rigby added a comment - Re-running same workload with https://review.couchbase.org/c/platform/+/167243 , pending_destruction_bytes is significantly lower - a few 10s of MB:

          People

            drigby Dave Rigby
            drigby Dave Rigby
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty