Spurious auto-failover possible if Magma compaction visits a TTL'd document which has already been deleted

Note: Only affects:

• 7.1.x - 7.1.4-MP2 and upwards
• 7.2.0
  (There's no "7.1.4-MP2" version in Jira, only 7.1.4, so cannot express this directly.)

The kv_ep_data_read_failed stat is used by ns_server to detect sustained disk errors and fail over the node.

We increment the stat when a read fails but also when we ask the storage engine for a document which does not exist.

https://github.com/couchbase/kv_engine/blob/5e46b22da75d32691cfe5ab3c4dd8a6bafed3184/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc#LL956C2-L956C2

We only do that for single document reads, and not for batched reads (queued BGFetches).

While this behaviour exists since 7.0.0, before 7.1.5,/7.2.0, BGFetches for expired documents were queued and used a different code path and so even if the document was not found, we wouldn't increment the stat.

However, as of MB-53898 with compaction_expiry_fetch_inline=true (default) we now use the single document read code path which does report no_such_key as a read failure.

Under Magma specifically, compaction can see a version of a document in an SST, which has already been expired and had it's tombstone purged away, when the document tombstones was seen another SST, resulting in no_such_key.

Given the stat is used to trigger failover, we want to only increment it when an actual IO failure occurs.

This issue has only been observed when using the Magma storage engine.