Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48923

KV-engine stuck in crash loop after <6.6.1 to 6.6.1+ upgrade and SyncWrites in use

    XMLWordPrintable

Details

    • Triaged
    • 1
    • Yes
    • KV 2021-Oct-21

    Description

      NOTE: 7.0+ is not affected by this issue

      During upgrade from <6.6.1 to 6.6.2+, a new stat is added and tracked in _local/vbstate to count the bytes in on-disk Prepares - see MB-42306.

      There is a bug in how the updates to this stat are handled during compaction - the value is not clamped to zero and hence can underflow - see https://github.com/couchbase/kv_engine/blob/v6.6.1/engines/ep/src/couch-kvstore/couch-kvstore.cc#L1325-L1327 :

              auto prepareBytes = json.find("on_disk_prepare_bytes");
              if (prepareBytes != json.end()) {
                  auto onDiskPrepareBytes =
                          std::stoull(prepareBytes->get<std::string>()) -
                          ctx.stats.prepareBytesPurged;
                  *prepareBytes = std::to_string(onDiskPrepareBytes);
                  updateVbState = true;
              }
      

      Note that if ctx.stats.prepareBytesPurged is greater than prepareBytes read from disk, the value written will underflow to a negative number - or very large positive if represented as a unsigned value.

      This issue was fixed in 7.0.0 upwards as part of http://review.couchbase.org/c/kv_engine/+/140801/4/engines/ep/src/couch-kvstore/couch-kvstore.cc#1585 when this code was merged into the master branch at the time. Unfortunately it wasn't flagged as a customer-impacting bug and hence was not tracked as affecting v.6.6.x.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-6.6.4-9924 contains kv_engine commit 93350e0 with commit message:
          MB-48923: Avoid underflow of on_disk_prepare_bytes during compaction

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.4-9924 contains kv_engine commit 93350e0 with commit message: MB-48923 : Avoid underflow of on_disk_prepare_bytes during compaction

          Build couchbase-server-7.0.3-7010 contains kv_engine commit 57e005f with commit message:
          MB-48923: Merge commit '93350e0d6' into cheshire-cat

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.3-7010 contains kv_engine commit 57e005f with commit message: MB-48923 : Merge commit '93350e0d6' into cheshire-cat

          Build couchbase-server-7.0.3-7010 contains kv_engine commit 93350e0 with commit message:
          MB-48923: Avoid underflow of on_disk_prepare_bytes during compaction

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.3-7010 contains kv_engine commit 93350e0 with commit message: MB-48923 : Avoid underflow of on_disk_prepare_bytes during compaction

          Build couchbase-server-7.1.0-1770 contains kv_engine commit 57e005f with commit message:
          MB-48923: Merge commit '93350e0d6' into cheshire-cat

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1770 contains kv_engine commit 57e005f with commit message: MB-48923 : Merge commit '93350e0d6' into cheshire-cat

          Build couchbase-server-7.1.0-1770 contains kv_engine commit 93350e0 with commit message:
          MB-48923: Avoid underflow of on_disk_prepare_bytes during compaction

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1770 contains kv_engine commit 93350e0 with commit message: MB-48923 : Avoid underflow of on_disk_prepare_bytes during compaction

          Validated using Enterprise Edition 6.6.5 build 10068.

          Closing the ticket.

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated using Enterprise Edition 6.6.5 build 10068. Closing the ticket.

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            drigby Dave Rigby
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty