Details
-
Improvement
-
Resolution: Fixed
-
Major
-
6.5.0
-
KV Sprint 2020-Oct
Description
Allow compaction to be partially concurrent with Flushing, by allowing the Flusher to continue to run while compacting the previous file. When compaction gets to the the end of the snapshot of the file when it opened, then block the Flusher and check if any more mutations have been written to the (old) couchstore file, if so copy just that (hopefully small) number over to the new file.
This allows the bulk (99+%) of the compaction to occur without blocking the flusher, minimising the period when exclusive access is required.
Attachments
Issue Links
- has to be done after
-
MB-41747 Couchstore: Memcached crashed in Warmup::loadKVPairsforShard(unsigned short) () from /opt/couchbase/bin/../lib/libep.so
- Closed
-
MB-40415 "_local/vbstate" may get out of sync if a crash happens right after compaction
- Closed
- relates to
-
MB-38426 Reduce SyncWrite latency
- Closed
For Gerrit Dashboard: MB-38428 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
132147,4 | MB-38428: Allow flusher to run in parallel with compaction WIP | master | kv_engine | Status: ABANDONED | 0 | -1 |
135954,28 | MB-38428: Allow compaction and flusher in parallel | master | kv_engine | Status: MERGED | +2 | +1 |
138388,3 | MB-38428: Run compaction and flush in parallel | master | kv_engine | Status: MERGED | +2 | +1 |
138896,3 | MB-42268: Revert "MB-38428: Run compaction and flush in parallel" | master | kv_engine | Status: MERGED | +2 | +1 |
138979,6 | MB-38428: Run compaction and flush in parallel | master | kv_engine | Status: ABANDONED | 0 | 0 |
139381,2 | MB-38428: Add more hooks for PiTR compaction and reply | master | couchstore | Status: MERGED | +2 | +1 |
139408,11 | MB-38428: Run flusher and compactor in parallel | master | kv_engine | Status: MERGED | +2 | +1 |
139465,4 | MB-38428: Remove extra validation code | master | kv_engine | Status: MERGED | +2 | +1 |
167485,4 | MB-49702: Revert "MB-18426: Reduce compaction_number_of_kv_workers from 4 to 1" | master | ns_server | Status: MERGED | +2 | +1 |