Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50461

Bgfetch modifies StoredValue without updating HashTable stats

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.0.0, 6.0.0, 5.1.3, 5.5.6, 6.5.2, 6.6.3, 7.0.2
    • 7.1.0
    • couchbase-bucket
    • Untriaged
    • 1
    • Unknown
    • KV 2022-Jan

    Description

      If a request tries to read/modify a deleted value (e.g., to access system xattrs) the item will be bgfetched from disk. It may subsequently be ejected - under value eviction, this leaves a deleted, non-resident StoredValue in the HashTable[1].

      The corresponding tombstone on disk may then be purged after the appropriate interval. This leaves a deleted, non resident SV which only exists in memory. If the same document is then requested again, another bgfetch will be triggered. This will find the item does not exist on disk.

      BGFetch code implicitly assumes that if an item does not exist on disk the value in the HT must be tempInitialItem (seqno = StoredValue::state_temp_init) placeholder[2]. Because of this assumption, it changes the existing StoredValue to temp+nonExistent (seqno = StoredValue::state_non_existent_key), without updating any stats (a transition from temp -> temp wouldn't change any stat values).

      As a result, the second bgfetch for the deleted, non-resident item will transition the item from non-temp -> temp, but does not update relevant stats in the HashTable. This leads to HashTable::Statistics::numItems being higher than it should be, and HashTable::Statistics::numTempItems being lower than it should be. numTempItems is a cb::NonNegativeCounter<size_t>, and will silently clamp at zero.

      Externally, This presents as curr_items climbing over time. To confirm this issue is present, curr_items can be compared to the sum of the active datatype counts:

      |ep_active_datatype_json| + |ep_active_datatype_json,xattr| + |ep_active_datatype_raw| + |ep_active_datatype_snappy| + |ep_active_datatype_snappy,json| + |ep_active_datatype_snappy,json,xattr| + |ep_active_datatype_snappy,xattr| + |ep_active_datatype_xattr|
      

      If curr_items is not equal to (more specifically, is greater than) this sum, this is a strong suggestion of this issue.

      It appears Sync Gateway is likely to trigger this issue through use of the _sync system xattr of deleted values.


      [1] MB-50423 addresses the fact that this metadata may then linger forever, and may change this behaviour.
      [2] Outside of MB-50423, this is true - any non-temp item in memory should also exist on disk.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-2147 contains kv_engine commit b470829 with commit message:
            MB-50461: [4/N] Simplify completeBGFetchForSingleItem

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2147 contains kv_engine commit b470829 with commit message: MB-50461 : [4/N] Simplify completeBGFetchForSingleItem

            Build couchbase-server-7.1.0-2147 contains kv_engine commit 8738b67 with commit message:
            MB-50461: [3/N] Remove duplication in completeBGFetchForSingleItem

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2147 contains kv_engine commit 8738b67 with commit message: MB-50461 : [3/N] Remove duplication in completeBGFetchForSingleItem

            Build couchbase-server-7.1.0-2147 contains kv_engine commit abd8cf0 with commit message:
            MB-50461: [2/N] Simplify completeBGFetchForSingleItem with early exits

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2147 contains kv_engine commit abd8cf0 with commit message: MB-50461 : [2/N] Simplify completeBGFetchForSingleItem with early exits

            Build couchbase-server-7.1.0-2147 contains kv_engine commit 5a99aaf with commit message:
            MB-50461: [1/N] Simplify completeBGFetchForSingleItem with early exits

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2147 contains kv_engine commit 5a99aaf with commit message: MB-50461 : [1/N] Simplify completeBGFetchForSingleItem with early exits

            Build couchbase-server-7.1.0-2156 contains kv_engine commit f1a5e2d with commit message:
            MB-50461: Ensure bgfetcher never changes non-temp item -> temp item

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2156 contains kv_engine commit f1a5e2d with commit message: MB-50461 : Ensure bgfetcher never changes non-temp item -> temp item

            Changes for MB-50423 resolve the only known route to this occurring, patches merged under this MB ensure that no other bugs can lead to the same stats issue.

            james.harrison James Harrison added a comment - Changes for MB-50423 resolve the only known route to this occurring, patches merged under this MB ensure that no other bugs can lead to the same stats issue.

            People

              james.harrison James Harrison
              james.harrison James Harrison
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty