Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57609

Spurious auto-failover possible if Magma compaction visits a TTL'd document which has already been deleted

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      Note: Only affects:

      • 7.1.x - 7.1.4-MP2 and upwards
      • 7.2.0
        (There's no "7.1.4-MP2" version in Jira, only 7.1.4, so cannot express this directly.)

      The kv_ep_data_read_failed stat is used by ns_server to detect sustained disk errors and fail over the node.

      We increment the stat when a read fails but also when we ask the storage engine for a document which does not exist.

      https://github.com/couchbase/kv_engine/blob/5e46b22da75d32691cfe5ab3c4dd8a6bafed3184/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc#LL956C2-L956C2

      We only do that for single document reads, and not for batched reads (queued BGFetches).

      While this behaviour exists since 7.0.0, before 7.1.5,/7.2.0, BGFetches for expired documents were queued and used a different code path and so even if the document was not found, we wouldn't increment the stat.

      However, as of MB-53898 with compaction_expiry_fetch_inline=true (default) we now use the single document read code path which does report no_such_key as a read failure.

      Under Magma specifically, compaction can see a version of a document in an SST, which has already been expired and had it's tombstone purged away, when the document tombstones was seen another SST, resulting in no_such_key.

      Given the stat is used to trigger failover, we want to only increment it when an actual IO failure occurs.

      This issue has only been observed when using the Magma storage engine.

       

      Issue Resolution
      A spurious auto-failover could happen when Magma compaction visited a TTL’d document that was already deleted. Document not found does not now increment the number of read failures.

      Attachments

        Issue Links

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              vesko.karaganev Vesko Karaganev
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty