Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57304

KV rebalance can hang if other DCP clients consume all backfill slots



    • Untriaged
    • 0
    • No
    • KV 2023-4


      As seen in rebalance test described in MB-57271, a cluster with KV plus other services (FTS and GSI in the above instance) performing a (KV) rebalance can hang if KV requires backfills and all of the available backfill slots (default 4096) are consumed by other services.

      For example, it was observed that 4231 streams were attempting to backfill:

      $ rg 'stream_\d+_state:' stats.log | rg backfilling|wc -l

      Which were made up of:

      $ rg 'stream_\d+_state:' stats.log | rg backfilling | cut -d : -f 2 | sort | uniq -c
      3420 fts
         4 replication
       807 secidx

      However given there's only 4096 possible at once, a number of stream were pending (waiting for a slot to become available before they can start):

      $ rg 'backfill_num_pending' stats.log | rg -v -w 0 | cut -d: -f 2,4- | column -t
      replication:default0:backfill_num_pending:  4
      secidx:backfill_num_pending:                8
      secidx:backfill_num_pending:                8
      secidx:backfill_num_pending:                6
      secidx:backfill_num_pending:                7
      secidx:backfill_num_pending:                17
      secidx:backfill_num_pending:                9
      secidx:backfill_num_pending:                20
      secidx:backfill_num_pending:                14
      secidx:backfill_num_pending:                16

      While we can also look at reducing the number of concurrent streams other services create, Ideally we want a solution such that KV is "defensive" - irrespective of what other services request, it can always make rebalance progress.


      Issue Resolution
      Data Service rebalance duration was significantly impacted if other DCP clients created a large number of Streams, if those streams needed to be read from disk, due to the lack of prioritizing between rebalance and other DCP clients. The number of backfills each DCP client can perform concurrently has been limited to allow fairer allocation of resources.


        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby (Inactive)
              0 Vote for this issue
              12 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes