Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49469

Open checkpoint might grow unbounded and lead to high allocation in key-index at replica

    XMLWordPrintable

Details

    • 1
    • Yes
    • KV 2021-Nov, KV 2021-Dec, KV 2022-Jan, KV 2022-Feb

    Description

      Repetition of the the same test as in MB-49134, this time on a toy-build that contains stats improvements by MB-48587.

      Run: http://perf.jenkins.couchbase.com/job/oceanus/7373/
      Build: 7.1.0-1670 + MB-48587

      The test is a simple bulk load (via restore) of 320 Million items on a 2-KVNode cluster. 1 Bucket, 1 Replica.
      We have a success from the test PoV, as all the items are successfully stored and replicated.
      The problem is the checkpoints_overhead_estimate_replica stat (yellow line) that shows a constantly increasing asymptotic trend.
      Charts show that the checkpoints_overhead_estimate_index_replica (dark-blue line) is what determines that trend.

      Note
      The above ran with the default checkpoint_memory_ratio=30%.
      Interestingly enough, lowering the CMQuota (eg, 10%, http://perf.jenkins.couchbase.com/job/oceanus/7372/parameters) shrinks all the other mem-usage stats too, so the pattern stays the same.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-49469
          # Subject Branch Project Status CR V

          Activity

            People

              paolo.cocchi Paolo Cocchi
              paolo.cocchi Paolo Cocchi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty