Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57609

Spurious auto-failover possible if Magma compaction visits a TTL'd document which has already been deleted



    • Untriaged
    • 0
    • Yes


      Note: Only affects:

      • 7.1.x - 7.1.4-MP2 and upwards
      • 7.2.0
        (There's no "7.1.4-MP2" version in Jira, only 7.1.4, so cannot express this directly.)

      The kv_ep_data_read_failed stat is used by ns_server to detect sustained disk errors and fail over the node.

      We increment the stat when a read fails but also when we ask the storage engine for a document which does not exist.


      We only do that for single document reads, and not for batched reads (queued BGFetches).

      While this behaviour exists since 7.0.0, before 7.1.5,/7.2.0, BGFetches for expired documents were queued and used a different code path and so even if the document was not found, we wouldn't increment the stat.

      However, as of MB-53898 with compaction_expiry_fetch_inline=true (default) we now use the single document read code path which does report no_such_key as a read failure.

      Under Magma specifically, compaction can see a version of a document in an SST, which has already been expired and had it's tombstone purged away, when the document tombstones was seen another SST, resulting in no_such_key.

      Given the stat is used to trigger failover, we want to only increment it when an actual IO failure occurs.

      This issue has only been observed when using the Magma storage engine.


      Issue Resolution
      A spurious auto-failover could happen when Magma compaction visited a TTL’d document that was already deleted. Document not found does not now increment the number of read failures.


        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              ashwin.govindarajulu Ashwin Govindarajulu
              vesko.karaganev Vesko Karaganev
              0 Vote for this issue
              8 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes