Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26681

[RocksDB] Allow to set the Thread Pool size independently from the per-DB 'max_background_jobs' setting

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Major
    • master
    • master
    • couchbase-bucket
    • None

    Description

      Flusher and Compaction tasks in RocksDB are per database (i.e vBucket), not global - however we initially thought they were. As such, the existing code to control the number of flushers/compactors always sets the number of background threads to the same as the number of tasks which run on those threads.

      That would be fine if we had a single RocksDB database - we would have:

      • F flusher tasks running on F high priority threads, and
      • C compaction tasks runnning on C high priority threads.

      However, given we have one DB per vBucket, this means we end up with:

      • F * V flusher tasks running on F threads, and
      • C * V compaction tasks running on C threads.

      Where V is the number of vBuckets.

      As a result we cannot configure optimally in either direction - we either end up with too many tasks scheduled across too many threads, or too few tasks scheduled over too few threads.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            paolo.cocchi Paolo Cocchi
            paolo.cocchi Paolo Cocchi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty