Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.5, 6.6.2, 6.5.2, 6.6.3, 7.0.2, 7.0.3, 7.0.4
-
Untriaged
-
1
-
Unknown
-
KV 2022-Jan, KV Oct 2022
Description
Deleted items may be bgfetched back into memory under value eviction buckets in response to a get with the option GET_DELETED_VALUE set.
This will load the metadata and value back into the hashtable as a deleted, resident, non-temporary item.
Later, the value may again be ejected by the ItemPager, but the metadata will never be removed, even after the tombstone is purged from disk.
A pathological workload could lead to the entire bucket quota being occupied by deleted item metadata despite having zero items on disk (not even tombstones). This memory would not be recoverable through normal mechanisms, but would be freed after destroying the owning vbucket (e.g., rebalancing the vb elsewhere, restarting memcached).
Storing a new value for the same key can still replace the delete metadata.
Clusters including Sync Gateway are likely to demonstrate this issue to some degree, as it appears SG may request deleted items to read system xattrs.
The number of deleted items in the hashtable is exposed as ht_num_deleted_items from 7.1.0 onward, or can be found at a per-vb level with mcstat ... _hash-dump <vbid>.
Full eviction buckets are not significantly impacted, as the delete metadata can be ejected by the ItemPager.
This issue appears to date back to at least 5.0.X
Attachments
Issue Links
- relates to
-
MB-50425 Subdoc: Race in checking for already-existing Deleted document with Add|CreateAsDeleted
- Closed
-
MB-52793 Replica can persist deletion with xattr datatype but empty value
- Closed
-
MB-51424 KVBucket::getValue can lead to incorrectly incrementing "failure_get" a failover trigger stat
- Closed