Ephemeral auto_delete bucket can trigger pager constantly with no memory reduction.

This bug is for ephemeral with auto_delete.

There is a mismatch in the quantities used to first trigger memory recovery and second to stop memory recovery. A bucket can be sized and utilised in such a way that the memory recovery is constantly triggered only to cancel, leading to noticeable CPU usage and a lack of any memory reduction. In this case memory recovery is the ItemPager.

The problem exists in that functions used in the trigger code have persistent vs ephemeral implementations, with the ephemeral code deriving adjust values that aim to reflect active memory only (because replicas cannot be included in memory recovery). Later in the ItemPager unadjusted quantities are used leading to early termination with 0 memory recovered.

• The trigger code in checkAndMaybeFreeMemory uses two virtual methods in the condition https://src.couchbase.org/source/xref/7.1.4/kv_engine/engines/ep/src/kv_bucket.cc?r=290d2081&fi=visitAsync#2162
• The early termination code uses unadjusted values https://src.couchbase.org/source/xref/7.1.4/kv_engine/engines/ep/src/paging_visitor.cc?r=0aad58ba#194-201

In the case which triggered this MB a 27 node cluster has a small 110MiB quota ephemeral bucket with 38 replica and 37 active vbuckets (total of 75)

• high-water mark = 93.5MiB
  • getPageableMemHighWatermark adjusted high-water mark = 46.12MiB
• low-water mark = 82.5MiB
  • getPageableMemLowWatermark adjusted low-water mark = 40.7MiB
• mem_used = 78MiB
  • getPageableMemCurrent harder confirm as we don't track the two replica memory values used in the implementation. However it has to be 46.12MiB or higher to trigger ItemPager

So if mem_used sits at 78MiB it is enough to trigger the ItemPager but less than the low-water mark so pager does nothing.

This issue is reprocubile if we recreate something simialr to the above. In this case trinity code was used (cluster_run).

• COUCHBASE_NUM_VBUCKETS=60
• cluster_run -n2
• 1x ephemeral bucket with auto_delete and 110MiB quota.
• Use pillow fight to keep inserting items until mem_used reaches ~78MiB (e.g 81,788,928 bytes)

Reproduction can be noted by tracking the ep_num_pager_runs statistic, it is increasing fast.