Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54975

Ephemeral replica memory stats not tracked correctly when deleting replica vBucket

    XMLWordPrintable

Details

    • Triaged
    • 0
    • Unknown
    • KV 2023-2

    Description

      replicaHTMemory and replicaCheckpointOverhead are not tracked correctly when deleting a replica vBucket via the DEL_VBUCKET path as no call is made to EphemeralVBucket::updateStatsForStateChange(); the function is only called in the SET_VBUCKET path.

      The issue results in incorrect memory tracking of replica vBuckets in Ephemeral Buckets which can cause the item pager to skip paging out anything as it things memory usage is already low enough. Whilst the issue was introduced with the fix for MB-41804, it appears to manifest in the same way (sustained tmp_ooms during workload), although it's unclear if we're more or less likely to hit this issue (I suspect the answer is it depends).

      We can diagnose this issue by observing that the ItemPager is attempting to run (trace dump likely to be the best method of doing this but we can also look at the Task runtime/schedule histogram (total counts)) but the pager_runs stat is not increasing, and then by calculating the "pageable" memory values using the memory stats and vBucket counts to determine if we are aborting the pager run due to being under the pageable low watermark.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ben.huddleston Ben Huddleston
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty