Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49469

Open checkpoint might grow unbounded and lead to high allocation in key-index at replica

    XMLWordPrintable

Details

    • 1
    • Yes
    • KV 2021-Nov, KV 2021-Dec, KV 2022-Jan, KV 2022-Feb

    Description

      Repetition of the the same test as in MB-49134, this time on a toy-build that contains stats improvements by MB-48587.

      Run: http://perf.jenkins.couchbase.com/job/oceanus/7373/
      Build: 7.1.0-1670 + MB-48587

      The test is a simple bulk load (via restore) of 320 Million items on a 2-KVNode cluster. 1 Bucket, 1 Replica.
      We have a success from the test PoV, as all the items are successfully stored and replicated.
      The problem is the checkpoints_overhead_estimate_replica stat (yellow line) that shows a constantly increasing asymptotic trend.
      Charts show that the checkpoints_overhead_estimate_index_replica (dark-blue line) is what determines that trend.

      Note
      The above ran with the default checkpoint_memory_ratio=30%.
      Interestingly enough, lowering the CMQuota (eg, 10%, http://perf.jenkins.couchbase.com/job/oceanus/7372/parameters) shrinks all the other mem-usage stats too, so the pattern stays the same.

      Attachments

        1. 1670-enh_cm-mem.png
          414 kB
          Paolo Cocchi
        2. MB-49469_cm_mem_drops.png
          190 kB
          Paolo Cocchi
        3. MB-49469_cm-active.png
          246 kB
          Paolo Cocchi
        4. MB-49469_cm-replica.png
          281 kB
          Paolo Cocchi
        5. MB-49469_max-10_multiplier-10_cm-mem-state.png
          401 kB
          Paolo Cocchi
        6. MB-49469_max-10_multiplier-10.png
          516 kB
          Paolo Cocchi
        7. MB-49469_max-10_multiplier-100_cm-mem-state.png
          288 kB
          Paolo Cocchi
        8. MB-49469_max-10_multiplier-100.png
          460 kB
          Paolo Cocchi
        9. MB-49469_unlimited_max_checkpoints.png
          195 kB
          Paolo Cocchi

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              paolo.cocchi Paolo Cocchi
              paolo.cocchi Paolo Cocchi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty