Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56011

Magma compaction increases front-end GET latency at high ops/sec at low resident ratio



    • Untriaged
    • 0
    • Unknown


      We've noticed that at low resident ratios and high ops/sec (with GETS and SETS), magma compaction seems to impact front-end GET latency (and, although we don't have empirical data for this, it is expected that durable writes with persistence would also be affected).

      As far as I can tell, this isn't a regression but something that's been around since Magma was first introduced. It also isn't something that appears to have been getting worse over time.

      The way we concluded this is contained within this slack discussion: https://couchbase.slack.com/archives/CF4FTNFEC/p1678898374286419

      Essentially, I had first notice flusher dedupe spikes which correlated with (and were caused by) disk write queue spikes. These disk write queue spikes in turn looked to be caused by something slowing down writes to disk, which looked like it was compaction (specifically, amount of bytes read/written during compaction). From here we saw that bg_fetch time was also impacted, and from this we verified that front-end get latency was also noticeably impacted (as well as ops/sec, since we use a sequential workload generator).


      So far I can only find this behaviour in tests which are at low (single digit) resident ratios and which are pushing high ops/sec, where "high" is relative to the hardware's capabilities. Some examples of tests where we see this kind of behaviour (admittedly its sometimes quite subtle):


      I took the cbmonitor images from the report from this 7.1.4 throughput test: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_714-3601_access_key_prefix_a2b8


      When we compare 7.1.4 to 7.2.0 for that particular test, we see about the same impact in both: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_720-5241_access_key_prefix_0d08&snapshot=rhea_714-3601_access_key_prefix_a2b8


        No reviews matched the request. Check your Options in the drop-down menu of this sections header.



            owend Daniel Owen
            Daniel.nagy Daniel Nagy
            0 Vote for this issue
            2 Start watching this issue



              Gerrit Reviews

                There are no open Gerrit changes