Due to a change in eviction behaviour in 7.1.0 ( https://review.couchbase.org/c/kv_engine/+/159487 ), we stopped removing tempDeleted values from the HashTable when the ExpiryPager ran.
This was done with a good reason: Pre-change, we would bump up expired item stats, which would then be misleading because these temp items are not user documents. The idea was to only clean them up by the ItemPager, when we hit the high watermark.
However, the unintended consequence is that we can get significant memory usage by tempDeleted items, which while benign (we can evict these), is misleading to the customer, who only sees that mem_used is growing.
The proposed solution is to revert the old behaviour, clean up those items in the ExpiryPager (which runs every 10 mins by default), but to do so inline, without queuing them in the expired items list and without incrementing the expired item stats.
Note that technically, a frequently accessed tempDeleted item can have a higher value than a very cold valued item. In a sense, the old behaviour was closer to providing a globally optimal solution that reduces bg fetches. However, because memory usage by tempDeleted items is not directly observable, we'd like to avoid filling up the bucket quota with them.