Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37680

Order DCP streams when streaming from multiple VB

    XMLWordPrintable

Details

    Description

      Problem

      For consistency cbbackupmgr is forced to open all vbucket streams at once.  This a has a performance impact because the streams are interlaced, which causes cbbackupmgr to jump between different vbucket files.

      For supporting backup to S3 it is too expensive to hold all 1024 vbuckets open at once, because to be cost efficient the upload chunks have to be large.

      A possible solution is to open and stream one vbucket at a time, however this causes consistency problems.

      It is suggested that a DCP control message cold be used to state that it will open the file handle to all vbuckets, which gives consistency, however stream only one vbucket at a time

       

       

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-40052 DCP producer not backfilling data after previously reaching maxBackfills limit

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-39653 [CBM] cbbackupmgr failed to backup data

          • Test Blocker - QE-related tests can't proceed
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-39683 Backup performance tests failed on 7.0.0-2208

          • Test Blocker - QE-related tests can't proceed
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-37681 Handle disconnect during an initial complete DCP stream

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MB-39529 Modifiy the DCP client so that we can utilise sequential backfilling

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-37680
          # Subject Branch Project Status CR V
          131705,5 MB-31972: Remove BackfillManager::pendingBackfills master kv_engine Status: NEW -1 -1

          Activity

            People

              owend Daniel Owen
              owend Daniel Owen
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty