Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50291

Ephemeral memory recovery runs constantly if no active vBuckets defined

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • KV 2022-Jan

    Description

      When investigating MB-48070, it was observed that the Ephemeral memory recovery tasks (Eph tombstone hashtable cleaner, Eph tombstone stale item deleter) are running constantly during a number of ep_testsuite_dcp tests.

      Investigation found that these tests only have a single vBucket in the replica state (no active vBuckets). When checking to see if memory should be recovered (via KVBucket::checkAndMaybeFreeMemory), the lack of any active vBuckets causes EphemeralBucket::getPageableMemHighWatermark() to return 0 - i.e. indicating that memory recovery should be attempted if there is any memory used:

      // Trigger memory reduction (ItemPager) if we've exceeded the pageable high
      // watermark.
      void KVBucket::checkAndMaybeFreeMemory() {
          if (getPageableMemCurrent() > getPageableMemHighWatermark()) {
              attemptToFreeMemory();
          }
      }
      

      When attemptToFreeMemory() is called and the aforementioned memory recovery tasks are run, they essentially find nothing to free. However the next call to checkAndMaybeFreeMemory results in the same - i.e. we end up pointlessly consuming NonIO threads attempting to recover memory when there is nothing to do.

      Note while this scenario (zero active, >0 replica) vBuckets is unlikely to occur in a real system, it is possible during a failover scenario.

      Additionally this scenario occurs frequently in unit tests; resulting in tests taking much longer than necessary (particulary under sanitizers).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            drigby Dave Rigby (Inactive)
            drigby Dave Rigby (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty