Details
-
Improvement
-
Resolution: Fixed
-
Major
-
6.5.0
-
KV Sprint 2020-April
Description
Problem
For consistency cbbackupmgr is forced to open all vbucket streams at once. This a has a performance impact because the streams are interlaced, which causes cbbackupmgr to jump between different vbucket files.
For supporting backup to S3 it is too expensive to hold all 1024 vbuckets open at once, because to be cost efficient the upload chunks have to be large.
A possible solution is to open and stream one vbucket at a time, however this causes consistency problems.
It is suggested that a DCP control message cold be used to state that it will open the file handle to all vbuckets, which gives consistency, however stream only one vbucket at a time
Attachments
Issue Links
- causes
-
MB-40052 DCP producer not backfilling data after previously reaching maxBackfills limit
-
- Closed
-
Gerrit Reviews
| For Gerrit Dashboard: MB-37680 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 131705,5 | MB-31972: Remove BackfillManager::pendingBackfills | master | kv_engine | Status: NEW | -1 | -1 |
| 127873,5 | MB-37680: BackfillManager::backfill() - Simplify snooze logic | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 127881,11 | MB-37680: Introduce DCPBackfillIface | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 127997,2 | MB-37680: Enable Backfill tests for Ephemeral | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 128034,5 | MB-37680: Simplify Backfill finishing logic | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 128109,7 | MB-37680: MockActiveStream: default common arguments | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 128291,3 | Merge branch 'couchbase/mad-hatter' into 'couchbase/master' | master | kv_engine | Status: MERGED | +2 | +1 |
| 128375,4 | MB-37680: Add support for sequential DCP Backfill | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 129163,2 | Merge branch mad-hatter | master | kv_engine | Status: MERGED | +2 | +1 |
| 129288,2 | MB-37680: Reduce differences between eph and persistent backfill | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 129292,5 | MB-37680: Schedule backfill of snap_markers ASAP | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
| 129405,1 | Merge branch mad-hatter | master | kv_engine | Status: MERGED | +2 | +1 |
| 129425,2 | Merge branch mad-hatter | master | kv_engine | Status: MERGED | +2 | +1 |
| 129608,1 | Merge branch mad-hatter | master | kv_engine | Status: MERGED | +2 | +1 |
| 129910,3 | Merge branch mad-hatter into master | master | kv_engine | Status: MERGED | +2 | +1 |
| 129937,5 | Merge branch mad-hatter into master | master | kv_engine | Status: MERGED | +2 | +1 |
| 130995,3 | MB-40052 [3/3]: Account for Backfills in initializingQ on destruction | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |