Details
-
Bug
-
Resolution: Unresolved
-
Major
-
5.5.0
-
Triaged
-
No
Description
It has been observed that long running backfills (e.g. due to slow backup clients) can result in high disk space utilization as compaction occurs.
DCP maintains an open file handle to one version of a vBucket file until the backfill from the file is complete. If the backfill is slow, compaction could occur during the backfill, resulting in 2 copies of the vBucket that are persisted until the backfill completes.
In the event that all vBuckets are being backfilled concurrently, this can result in a doubling of disk space utilization. Further compactions can incrementally compound the disk space utilization.
While this is the expected mode of operation for DCP, a typical user wouldn't anticipate this to be a potential side effect of common operational tasks (like backups). The issue is also hard to diagnose as the old vBucket file is deleted and is only observable via lsof.
Ideally, we would be able to manage the disk space utilization more gracefully on the server side, or have our tools be more conservative with the number of concurrent backfills per node.