tempDeleted items cannot be evicted on value eviction buckets

tempDeleted items can be created on value eviction buckets for GET_METAs. The usual pageOut path will not remove those tempDeleted items from the HashTable for Value eviction buckets.

This can result in (potentially) the entire bucket quota being occupied by these, at which point customers will see 0% residency ratios and failing BG fetches.

Diagnosis
Value-eviction buckets which perform GET_META operations which return on-disk tombstones are potentially affected. The main user of GET_META is XDCR, which issues GET_META requests against the destination of a replication stream as part of conflict resolution. To encounter the issue the GET_META requests must find a tombstone (marker for previously deleted document), which would typically occur in a bi-directional XDCR relationship where the workload includes deleted documents.
An increasing value of kv_curr_temp_items - which does not significantly reduce periodically when the Expiry pager runs (every 10 minutes) is indicative of this issue.

Workaround
Change the eviction setting from value-eviction to full-eviction on the affected bucket(s). Note this required a restart of the Bucket, so will result in bucket going offline while it warms up.