Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25509

Periodically call fsync() when writing large amounts of data

    XMLWordPrintable

Details

    • Release Note

    Description

      When writing large amounts of data (e.g during compaction), read latency can be adversely affected if a single sync() is made at the end of writing all the data; as the IO subsystem has a large amount of outstanding writes to flush to disk. By issuing periodic sync() calls the affect on read latency can be signifcantly reduced.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            dhaikney David Haikney added a comment - - edited

            Image shows impact on bgwait of various different fsync thresholds. Tested on 4.5.1 toy build. Blue line denotes threshold change as do the markers on the graph at the correspond positions following levels: 0-1 == Off, 2-3 == 16M, 3-4 == 128K, 4-5 == 1M, >5 == 16M

            dhaikney David Haikney added a comment - - edited Image shows impact on bgwait of various different fsync thresholds. Tested on 4.5.1 toy build. Blue line denotes threshold change as do the markers on the graph at the correspond positions following levels: 0-1 == Off, 2-3 == 16M, 3-4 == 128K, 4-5 == 1M, >5 == 16M

            Build 5.1.0-1096 contains couchstore commit ea89a54399ec243e78d14397ca15fbad71550e87 with commit message:
            MB-25509: Add periodic sync()ing after N bytes written
            https://github.com/couchbase/couchstore/commit/ea89a54399ec243e78d14397ca15fbad71550e87

            build-team Couchbase Build Team added a comment - Build 5.1.0-1096 contains couchstore commit ea89a54399ec243e78d14397ca15fbad71550e87 with commit message: MB-25509 : Add periodic sync()ing after N bytes written https://github.com/couchbase/couchstore/commit/ea89a54399ec243e78d14397ca15fbad71550e87

            Build 5.1.0-1125 contains kv_engine commit a52dad0f1e993d1aa9f9b5b6646ba985d2fc3efc with commit message:
            MB-25509: Support calling fsync() periodically during compaction
            https://github.com/couchbase/kv_engine/commit/a52dad0f1e993d1aa9f9b5b6646ba985d2fc3efc

            build-team Couchbase Build Team added a comment - Build 5.1.0-1125 contains kv_engine commit a52dad0f1e993d1aa9f9b5b6646ba985d2fc3efc with commit message: MB-25509 : Support calling fsync() periodically during compaction https://github.com/couchbase/kv_engine/commit/a52dad0f1e993d1aa9f9b5b6646ba985d2fc3efc

            Build 5.1.0-1125 contains couchstore commit 745729c443265111e893bc8d82dc566192a71e55 with commit message:
            MB-25509: Add couchstore_encode_periodic_sync_flags()
            https://github.com/couchbase/couchstore/commit/745729c443265111e893bc8d82dc566192a71e55

            build-team Couchbase Build Team added a comment - Build 5.1.0-1125 contains couchstore commit 745729c443265111e893bc8d82dc566192a71e55 with commit message: MB-25509 : Add couchstore_encode_periodic_sync_flags() https://github.com/couchbase/couchstore/commit/745729c443265111e893bc8d82dc566192a71e55
            drigby Dave Rigby added a comment -

            For Release Notes:

            The Data Service Compaction task has been tuned to incrementally flush the new compacted file to disk. This significantly reduces the impact that compaction has on the disk read latencies for lookups which miss the managed cache (10x reduction in 99.9% GET latency).

            drigby Dave Rigby added a comment - For Release Notes: The Data Service Compaction task has been tuned to incrementally flush the new compacted file to disk. This significantly reduces the impact that compaction has on the disk read latencies for lookups which miss the managed cache (10x reduction in 99.9% GET latency).

            reopening temporarily to create a sub-task to backport from vulcan to 5.1

            dhaikney David Haikney added a comment - reopening temporarily to create a sub-task to backport from vulcan to 5.1

            People

              dhaikney David Haikney
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty