Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38173

Correct tracking of checkpoint_memory_overhead

    XMLWordPrintable

Details

    • Triaged
    • Unknown
    • KV Sprint 2020-April

    Description

      Discovered whilst investigating MB-38012 that we are currently tracking checkpoint memory overhead incorrectly. We appear to track the allocations of the checkpoint_index map, but not of the keys used in them.

      Consider the case where we have a large disk checkpoint that is entirely non-resident in the checkpoint (i.e. everything has been expelled) then the keys are a substantial part of this. In the case of MB-38012 we are tracking approx 80 bytes overhead per item, but this does not include the keys which are ~48 bytes. We were tracking 27GB memory overhead in the checkpoint but in reality this is more like 40GB.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          This is the problematic code (in Checkpoint)

              size_t getMemoryOverhead() const {
                  // All 3 indexes (preparedKey, committedKey and metaKey) share the
                  // same allocator and therefore getting the bytes allocated for the
                  // one will include the others.
                  return sizeof(Checkpoint) +
                         *(committedKeyIndex.get_allocator().getBytesAllocated()) +
                         *(toWrite.get_allocator().getBytesAllocated());
              }
          

          ben.huddleston Ben Huddleston added a comment - This is the problematic code (in Checkpoint) size_t getMemoryOverhead() const { // All 3 indexes (preparedKey, committedKey and metaKey) share the // same allocator and therefore getting the bytes allocated for the // one will include the others. return sizeof(Checkpoint) + *(committedKeyIndex.get_allocator().getBytesAllocated()) + *(toWrite.get_allocator().getBytesAllocated()); }

          Build couchbase-server-6.6.0-7589 contains kv_engine commit bbaf3ba with commit message:
          MB-38173: Track key allocations as part of checkpoint overhead

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7589 contains kv_engine commit bbaf3ba with commit message: MB-38173 : Track key allocations as part of checkpoint overhead

          Build couchbase-server-7.0.0-2030 contains kv_engine commit bbaf3ba with commit message:
          MB-38173: Track key allocations as part of checkpoint overhead

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2030 contains kv_engine commit bbaf3ba with commit message: MB-38173 : Track key allocations as part of checkpoint overhead

          Closing this ticket based on the unit_test results.

          Commit ref for kv_engine: ee5212ee2faf739a70a342ade60a64eb0191e421

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Closing this ticket based on the unit_test results. Commit ref for kv_engine: ee5212ee2faf739a70a342ade60a64eb0191e421

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ben.huddleston Ben Huddleston
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty