Details
-
Improvement
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
4.5.1, 4.6.3, 5.0.0
-
Release Note
Description
When writing large amounts of data (e.g during compaction), read latency can be adversely affected if a single sync() is made at the end of writing all the data; as the IO subsystem has a large amount of outstanding writes to flush to disk. By issuing periodic sync() calls the affect on read latency can be signifcantly reduced.
Attachments
Issue Links
- links to
Gerrit Reviews
Activity
Build 5.1.0-1096 contains couchstore commit ea89a54399ec243e78d14397ca15fbad71550e87 with commit message:
MB-25509: Add periodic sync()ing after N bytes written
https://github.com/couchbase/couchstore/commit/ea89a54399ec243e78d14397ca15fbad71550e87
Build 5.1.0-1125 contains kv_engine commit a52dad0f1e993d1aa9f9b5b6646ba985d2fc3efc with commit message:
MB-25509: Support calling fsync() periodically during compaction
https://github.com/couchbase/kv_engine/commit/a52dad0f1e993d1aa9f9b5b6646ba985d2fc3efc
Build 5.1.0-1125 contains couchstore commit 745729c443265111e893bc8d82dc566192a71e55 with commit message:
MB-25509: Add couchstore_encode_periodic_sync_flags()
https://github.com/couchbase/couchstore/commit/745729c443265111e893bc8d82dc566192a71e55
For Release Notes:
The Data Service Compaction task has been tuned to incrementally flush the new compacted file to disk. This significantly reduces the impact that compaction has on the disk read latencies for lookups which miss the managed cache (10x reduction in 99.9% GET latency).
reopening temporarily to create a sub-task to backport from vulcan to 5.1
Image shows impact on bgwait of various different fsync thresholds. Tested on 4.5.1 toy build. Blue line denotes threshold change as do the markers on the graph at the correspond positions following levels: 0-1 == Off, 2-3 == 16M, 3-4 == 128K, 4-5 == 1M, >5 == 16M