Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37205

[Jepsen] Couchbase crash during swap-rebalance nemesis due to an item being uncommitted in processSet()

    XMLWordPrintable

Details

    • Triaged
    • No
    • KV Sprint 2019-12

    Description

      During kv-engine-jepsen-nightly-351 we observed a and crash during the following test:
      lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=2 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level
      Crash Message:

      2019-12-07T06:07:09.270558+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc: 3180
      

      vbucket.cc:3180

              Expects(itm.isCommitted());
              getPassiveDM().completeSyncWrite(
                      itm.getKey(),
                      PassiveDurabilityMonitor::Resolution::Commit,
                      v->getBySeqno() /* prepareSeqno */);
      

      Couchbase version 6.5.1-6000

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby added a comment -

            Expect fix to be merged today (just waiting on CV).

            drigby Dave Rigby added a comment - Expect fix to be merged today (just waiting on CV).

            Build couchbase-server-6.5.0-4940 contains kv_engine commit ec90c05 with commit message:
            MB-37205: Allow duplicate prepares on Disk snapshots

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-4940 contains kv_engine commit ec90c05 with commit message: MB-37205 : Allow duplicate prepares on Disk snapshots

            Build couchbase-server-6.5.1-6005 contains kv_engine commit ec90c05 with commit message:
            MB-37205: Allow duplicate prepares on Disk snapshots

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6005 contains kv_engine commit ec90c05 with commit message: MB-37205 : Allow duplicate prepares on Disk snapshots

            Build couchbase-server-7.0.0-1133 contains kv_engine commit ec90c05 with commit message:
            MB-37205: Allow duplicate prepares on Disk snapshots

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1133 contains kv_engine commit ec90c05 with commit message: MB-37205 : Allow duplicate prepares on Disk snapshots

            Closing a we no longer see this crash on the Jepsen CV jobs after the fixes were committed.

            richard.demellow Richard deMellow added a comment - Closing a we no longer see this crash on the Jepsen CV jobs after the fixes were committed.

            People

              richard.demellow Richard deMellow
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty