Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50169

[Jepsen] Precondition failure: 'highestPrepareSeqno == s->highPreparedSeqno.it->getBySeqno()'

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • KV 2021-Dec

    Description

      The GSL Precondition failure: 'highestPrepareSeqno == s->highPreparedSeqno.it->getBySeqno()' has been observed during a Jepsen post-commit. Due to the highest prepare seqno not being equal to state's highPreparedSeqno seqno.

      PassiveDurabilityMonitor::PassiveDurabilityMonitor(
              VBucket& vb,
              int64_t highPreparedSeqno,
              int64_t highCompletedSeqno,
              std::vector<queued_item>&& outstandingPrepares)
          : PassiveDurabilityMonitor(vb, highPreparedSeqno, highCompletedSeqno) {
          auto s = state.wlock();
          int64_t highestPrepareSeqno = -1;
          for (auto& prepare : outstandingPrepares) {
              Expects(highestPrepareSeqno < prepare->getBySeqno());
              highestPrepareSeqno = prepare->getBySeqno();
              s->trackedWrites.emplace_back(std::move(prepare));
       
              // Advance the highPreparedSeqno iterator to point to the highest
              // SyncWrite which has been prepared.
              auto lastIt = std::prev(s->trackedWrites.end());
              if (lastIt->getBySeqno() <= highPreparedSeqno) {
                  s->highPreparedSeqno.it = lastIt;
              }
       
              // Advance the highCompletedSeqno iterator to point to the highest
              // SyncWrite which has been completed.
              //
              // Note: One might assume that this would always point to
              // trackedWrites.begin(), given that we are a newly minted PassiveDM and
              // hence would only be tracking incomplete SyncWrites. However, we
              // _could_ have been converted from an ActiveDM with null topology which
              // itself was converted from a previous PassiveDM which _did_ have
              // completed SyncWrites still in trackedWrites (because they haven't
              // been persisted locally yet).
              if (lastIt->getBySeqno() <= highCompletedSeqno) {
                  s->highCompletedSeqno.it = lastIt;
              }
          }
          if (!outstandingPrepares.empty()) {
              if (s->highPreparedSeqno.it != s->trackedWrites.end()) {
                  Expects(highestPrepareSeqno ==
                          s->highPreparedSeqno.it->getBySeqno()); // <---  FAILING LINE
              }
              if (s->highCompletedSeqno.it != s->trackedWrites.end()) {
                  Expects(highestPrepareSeqno >
                          s->highCompletedSeqno.it->getBySeqno());
              }
          }
      }
      

      The Jepsen results are attached here: 20211217T170205.000-0800.zip or can be viewed at http://172.23.120.13:8081/files/Couchbase-jenkins-kv-engine-jepsen-post-commit-5445/20211217T170205.000-0800

      It's possible this expects is missing some valid state of on disk, but it's hard to tell without investigating further, hence it not being a critical. Also the expects where added to help debug MB-49767

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              richard.demellow Richard deMellow
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty