Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40267

ActiveStreamCheckpointProcessorTask should not be a AuxIO task

    XMLWordPrintable

Details

    • Triaged
    • 1
    • No

    Description

      The ActiveStreamCheckpointProcessorTask is incorrectly assigned to the AuxIO thread pool (which is intended for IO-bound tasks not directly related to reading / flushing data).

      This bug traces back to when ActiveStreamCheckpointProcessorTask was created on the 3.x branch - from the review comments of the original patch (http://review.couchbase.org/c/ep-engine/+/57148/5/src/dcp-producer.cc#152):

      Jim Walker
      2015-11-24
      The initial reason is to try and distribute the load. NONIO is already running the DcpConsumer Processor(s), itempager, expirypager, checkpoint remover, hashtable resizer and many more (http://src.couchbase.org/source/search?q=&defs=&refs=NONIO_TASK_IDX&path=&hist=&type=&project=3.1.1) whereas AUXIO is running DcpBackfill and access scanner (other tasks appear to be TAP related). As stated in the comments, for future (master branch) this should be revisited, maybe creating a DCP task type or increasing NONIO threads?

      There is arguably two problems with it running AuxIO:

      1. AuxIO tasks can often have long runtimes (latencies) as they are performing synchronous disk IO - and hence could impact the scheduling latency of the ActiveStreamCheckpointProcessorTask which should ideally have low latency (given it directly affects overall replication latency).
      2. The AuxIO thread pool is small (compared to the NonIO thread pool) - thread counts for a few different machine sizes:
        #CPUs #AuxIO threads #NonIO threads
        8 1 2
        16 2 4
        32 4 8

        As such, DCP streaming (for example for in-memory phase of rebalance) can be limited in how much CPU it can use and be prematurely CPU-bound.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            drigby Dave Rigby (Inactive)
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty