Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36370

HiDD: Scalable DCP and disk backfill performance

    XMLWordPrintable

Details

    • KV: DCP Scalability
    • To Do

    Description

      We have observed that DCP throughput per server in a two-node cluster max out at 70K/s and 10K/s on disk backfill. This primarily becomes a bottleneck when the storage engine is faster than DCP. Since replica depends on DCP for replication, overall write throughput of the system is capped by DCP replication throughput.

      Attachments

        1. 25 - dcp_replica.png
          25 - dcp_replica.png
          246 kB
        2. 26 - mem_usage.png
          26 - mem_usage.png
          401 kB
        3. 26 - disk-queue fill-drain.png
          26 - disk-queue fill-drain.png
          334 kB
        4. fill.perf.script
          2.31 MB

        Issue Links

          duplicates

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. MB-30023 Byte-based backfill

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-34733 Replication queue limiting magma throughput

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-38012 1% DGM Test: Write ops/s dropped from 230k/s to 400/s due to swapping

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-29325 Intra-cluster replication uses only a fraction of network and disk before bottlenecking

          • Major - Major loss of function.
          • Closed
          links to

          Web Link DCP - MEM

          For Gerrit Dashboard: MB-36370
          # Subject Branch Project Status CR V
          131939,9 MB-36370: Trace DCP master kv_engine Status: NEW 0 -1
          133206,2 MB-36370: Add the new DCP "backoff" stats in Connection master kv_engine Status: NEW -1 -1
          133499,37 MB-36370: Add DCP cluster_test perf suite master kv_engine Status: NEW 0 0
          134989,6 MB-36370: DCP Consumer just increases HT item count at DcpMutation master kv_engine Status: NEW 0 -1
          132012,2 MB-36370: Remove BackfillManager::bytesForceRead master kv_engine Status: MERGED +2 +1
          135423,7 MB-36370: Optimize the cluster_testapp replication proxy master kv_engine Status: MERGED +2 +1

          Activity

            People

              paolo.cocchi Paolo Cocchi
              sarath Sarath Lakshman
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are 4 open Gerrit changes

                  PagerDuty