Bgfetch modifies StoredValue without updating HashTable stats

If a request tries to read/modify a deleted value (e.g., to access system xattrs) the item will be bgfetched from disk. It may subsequently be ejected - under value eviction, this leaves a deleted, non-resident StoredValue in the HashTable[1].

The corresponding tombstone on disk may then be purged after the appropriate interval. This leaves a deleted, non resident SV which only exists in memory. If the same document is then requested again, another bgfetch will be triggered. This will find the item does not exist on disk.

BGFetch code implicitly assumes that if an item does not exist on disk the value in the HT must be tempInitialItem (seqno = StoredValue::state_temp_init) placeholder[2]. Because of this assumption, it changes the existing StoredValue to temp+nonExistent (seqno = StoredValue::state_non_existent_key), without updating any stats (a transition from temp -> temp wouldn't change any stat values).

As a result, the second bgfetch for the deleted, non-resident item will transition the item from non-temp -> temp, but does not update relevant stats in the HashTable. This leads to HashTable::Statistics::numItems being higher than it should be, and HashTable::Statistics::numTempItems being lower than it should be. numTempItems is a cb::NonNegativeCounter<size_t>, and will silently clamp at zero.

Externally, This presents as curr_items climbing over time. To confirm this issue is present, curr_items can be compared to the sum of the active datatype counts:

|ep_active_datatype_json| + |ep_active_datatype_json,xattr| + |ep_active_datatype_raw| + |ep_active_datatype_snappy| + |ep_active_datatype_snappy,json| + |ep_active_datatype_snappy,json,xattr| + |ep_active_datatype_snappy,xattr| + |ep_active_datatype_xattr|

If curr_items is not equal to (more specifically, is greater than) this sum, this is a strong suggestion of this issue.

It appears Sync Gateway is likely to trigger this issue through use of the _sync system xattr of deleted values.

[1] MB-50423 addresses the fact that this metadata may then linger forever, and may change this behaviour.
[2] Outside of MB-50423, this is true - any non-temp item in memory should also exist on disk.