Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26681

[RocksDB] Allow to set the Thread Pool size independently from the per-DB 'max_background_jobs' setting

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Major
    • master
    • master
    • couchbase-bucket
    • None

    Description

      Flusher and Compaction tasks in RocksDB are per database (i.e vBucket), not global - however we initially thought they were. As such, the existing code to control the number of flushers/compactors always sets the number of background threads to the same as the number of tasks which run on those threads.

      That would be fine if we had a single RocksDB database - we would have:

      • F flusher tasks running on F high priority threads, and
      • C compaction tasks runnning on C high priority threads.

      However, given we have one DB per vBucket, this means we end up with:

      • F * V flusher tasks running on F threads, and
      • C * V compaction tasks running on C threads.

      Where V is the number of vBuckets.

      As a result we cannot configure optimally in either direction - we either end up with too many tasks scheduled across too many threads, or too few tasks scheduled over too few threads.

      Attachments

        For Gerrit Dashboard: MB-26681
        # Subject Branch Project Status CR V

        Activity

          People

            paolo.cocchi Paolo Cocchi
            paolo.cocchi Paolo Cocchi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty