Details
-
Task
-
Resolution: Fixed
-
Major
-
master
Description
Summary
The initial disk snapshot that a DCP consumer receives is not put in checkpoint, but is instead put into a dedicated backfill queue (see backfill in vbucket.h). Additional disk snapshots such as those caused by cursor dropping do not use the queue, and instead are added to a checkpoint.
With the goal of better managing resources that are used during rebalance - we want to simplify the control flow and number of buffers that are used. Therefore we want to do away with the dedicated backfill queue which adds unnecessary complexity for no benefit - and instead just use checkpoints.
Details
In PassiveStream::processMarker we check to see if the marker is for a disk snapshot and if the associated vbucket has a highSeqNo of zero. If so we move the buckets into a backfilling state. In addition we set the open checkpoint ID to zero, to indicate that we are in backfilling state.
As the DCP consumer processes mutations it checks to see if the associated vbucket is in backfilling state, and if so adds the items to the dedicated backfill queue as opposed to the checkpoint.
If i processes deletions it calls deleteWithMeta() passing through whether the vbucket is in a backfilling state. This ultimately ends up in queueDirty where depending if in backfilling state adds to the backfill queue or the checkpoint.
if (isBackfillItem) {
|
queueBackfillItem(qi, generateBySeqno);
|
notifyCtx.notifyFlusher = true;
|
/* During backfill on a TAP receiver we need to update the snapshot
|
range in the checkpoint. Has to be done here because in case of TAP
|
backfill, above, we use vb.queueBackfillItem() instead of
|
vb.checkpointManager->queueDirty() */
|
if (generateBySeqno == GenerateBySeqno::Yes) {
|
checkpointManager->resetSnapshotRange();
|
}
|
} else {
|
notifyCtx.notifyFlusher =
|
checkpointManager->queueDirty(*this,
|
qi,
|
generateBySeqno,
|
generateCas,
|
preLinkDocumentContext);
|
notifyCtx.notifyReplication = true;
|
if (GenerateCas::Yes == generateCas) {
|
v.setCas(qi->getCas());
|
}
|
}
|
When a disk snapshot end is received we then move the vbucket out of backfilling state.
The approach is to remove the use of the backfill queue and just make use of the checkpoint like all other snapshots (including additional disk snapshots).
Attachments
For Gerrit Dashboard: MB-31664 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
100716,12 | MB-31664: Add ability to disable use of disk backfill queue | master | kv_engine | Status: MERGED | +2 | +1 |
100913,22 | MB-31664: Don't accept streamRequests when receiving first disk snapshot | master | kv_engine | Status: MERGED | +2 | +1 |
101275,2 | MB-31664: Switch over to not using backfill queue | master | kv_engine | Status: MERGED | +2 | +1 |
101684,4 | Revert "MB-31664: Switch over to not using backfill queue" | master | kv_engine | Status: MERGED | +2 | +1 |
105643,3 | MB-31664: Reapply - Switch over to not using backfill queue | master | kv_engine | Status: MERGED | +2 | +1 |
112525,9 | MB-31664: Completely remove VBucket backfill queue | master | kv_engine | Status: MERGED | +2 | +1 |