Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29928

Defragmenter: Only run when fragmentation is high

    XMLWordPrintable

Details

    • KV-Engine Sprint 2021 July

    Description

      While investigating / tuning the defragmenter for MB-29879 it was observed that the current defragmenter policy - where it runs constantly - means that we are aways paying a price for the defragmenter.

      Specifically, we need to carefully balance how often and how long the defragmenter runs:

      1. Too often (e.g. running with defragmenter_interval=0) means it essentially consumes a whole NonIO thread; as it "greedily" runs regardless of the current memory usage / fragmentation.
      2. Too infrequently, and it may not be able to keep up with high throughput workloads (as seen in MB-29879) and fail to defragment sufficiently.

      If we would instead only schedule the defragmenter when we believe there is useful work for it to do, then we could afford to be more aggressive with the interval value when we do decide to run it.

      For example, the defragmenter could be changed to only actually visit the Bucket if heap fragmentation (RSS / mem_used) exceeds a given threshold. It could run relatively aggressively then, but once the heap fragmentation drops we would stop visiting again.

      CC Daniel Owen Paolo Cocchi

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty