Details
-
Bug
-
Resolution: Unresolved
-
Major
-
7.6.0
-
None
-
Triaged
-
0
-
No
Description
A DCP client which ends up in backfill and that makes no progress with the stream (but is responding to DCP noop) risks the node running out of disk. The issue is that the a disk data is "locked" (e.g. a file is open) and that prevents that snapshot being released when compaction later occurs. In the worst case we could be tripling the disk usage, instead of double (when compacting).
E.g. for 1 vbucket.
- n GB for the file held open by backfill (file-v1)
- n GB for a new file being created by compaction (file-v3)
- n GB for the old file that compaction is released (file-v2)
KV can inspect the streams that are in backfill and apply some logic to detect no progress and secondly it could only apply the force-end when it knows the vbucket is growing - i.e. it's fine to hold the backfill if the system is doing nothing.
Attachments
Gerrit Reviews
For Gerrit Dashboard: MB-62703 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
214447,12 | MB-62703: Close DCP producer which has no streams | master | kv_engine | Status: NEW | 0 | -1 |
214579,13 | MB-62703: End backfilling (disk) streams which make no progress | master | kv_engine | Status: NEW | -1 | +1 |
214580,12 | MB-62703: End backfilling (memory) streams which make no progress | master | kv_engine | Status: NEW | -1 | -1 |
212662,4 | MB-62703: BySeqnoScanContext should own lastReadSeqno | master | kv_engine | Status: MERGED | +2 | +1 |
212672,4 | MB-62703: Remove DCPBackfillDisk class | master | kv_engine | Status: MERGED | +2 | +1 |
212880,2 | MB-62703: Track the "position" of a backfill | master | kv_engine | Status: ABANDONED | 0 | 0 |
214512,4 | MB-62703: Provide a KVBucket getter for Configuration | master | kv_engine | Status: MERGED | +2 | +1 |