tempDeleted items can be created on value eviction buckets for GET_METAs. The usual pageOut path will not remove those tempDeleted items from the HashTable for Value eviction buckets.
This can result in (potentially) the entire bucket quota being occupied by these, at which point customers will see 0% residency ratios and failing BG fetches.
Value-eviction buckets which perform GET_META operations which return on-disk tombstones are potentially affected. The main user of GET_META is XDCR, which issues GET_META requests against the destination of a replication stream as part of conflict resolution. To encounter the issue the GET_META requests must find a tombstone (marker for previously deleted document), which would typically occur in a bi-directional XDCR relationship where the workload includes deleted documents.
An increasing value of kv_curr_temp_items - which does not significantly reduce periodically when the Expiry pager runs (every 10 minutes) is indicative of this issue.
Change the eviction setting from value-eviction to full-eviction on the affected bucket(s). Note this required a restart of the Bucket, so will result in bucket going offline while it warms up.
|GET_META requests for deleted items fetched metadata in memory which was not evicted in value-eviction buckets.
|Metadata items are now cleaned when the expiry pager runs.