Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37205

[Jepsen] Couchbase crash during swap-rebalance nemesis due to an item being uncommitted in processSet()

    XMLWordPrintable

Details

    • Triaged
    • No
    • KV Sprint 2019-12

    Description

      During kv-engine-jepsen-nightly-351 we observed a and crash during the following test:
      lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=2 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level
      Crash Message:

      2019-12-07T06:07:09.270558+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc: 3180
      

      vbucket.cc:3180

              Expects(itm.isCommitted());
              getPassiveDM().completeSyncWrite(
                      itm.getKey(),
                      PassiveDurabilityMonitor::Resolution::Commit,
                      v->getBySeqno() /* prepareSeqno */);
      

      Couchbase version 6.5.1-6000

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            richard.demellow Richard deMellow created issue -
            owend Daniel Owen made changes -
            Field Original Value New Value
            Assignee Daniel Owen [ owend ] Ben Huddleston [ ben.huddleston ]
            owend Daniel Owen made changes -
            Fix Version/s Mad-Hatter [ 15037 ]
            owend Daniel Owen made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            richard.demellow Richard deMellow made changes -
            Summary [Jepsen] Couchbase crash during [Jepsen] Couchbase crash during swap-rebalance nemesis due to an item being uncommitted in processSet()
            owend Daniel Owen made changes -
            Sprint KV Sprint 2019-12 [ 939 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            richard.demellow Richard deMellow made changes -
            Remote Link This issue links to "20191206T215739.000-0800.zip (Web Link)" [ 18945 ]
            richard.demellow Richard deMellow made changes -
            Description During [kv-engine-jepsen-nightly-351|http://cv.jenkins.couchbase.com/view/Commit%20Validation/view/Jepsen/job/kv-engine-jepsen-nightly/351/] we observed a and crash during the following test:
            {{}}
            Crash Message:
            {noformat}
            2019-12-07T06:07:09.270558+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc: 3180
            {noformat}

            {code:cpp|title=vbucket.cc:3180}
                    Expects(itm.isCommitted());
                    getPassiveDM().completeSyncWrite(
                            itm.getKey(),
                            PassiveDurabilityMonitor::Resolution::Commit,
                            v->getBySeqno() /* prepareSeqno */);
            {code}
            Couchbase version 6.5.1-6000
            During [kv-engine-jepsen-nightly-351|http://cv.jenkins.couchbase.com/view/Commit%20Validation/view/Jepsen/job/kv-engine-jepsen-nightly/351/] we observed a and crash during the following test:
            {{lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=rebalance --scenario=swap-rebalance --no-autofailover --replicas=2 --disrupt-count=1 --kv-timeout=30 --durability=0:0:0:100 --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level}}
            Crash Message:
            {noformat}
            2019-12-07T06:07:09.270558+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc: 3180
            {noformat}

            {code:cpp|title=vbucket.cc:3180}
                    Expects(itm.isCommitted());
                    getPassiveDM().completeSyncWrite(
                            itm.getKey(),
                            PassiveDurabilityMonitor::Resolution::Commit,
                            v->getBySeqno() /* prepareSeqno */);
            {code}
            Couchbase version 6.5.1-6000
            owend Daniel Owen made changes -
            Due Date 10/Dec/19
            lynn.straus Lynn Straus made changes -
            Labels approved-for-mad-hatter
            lynn.straus Lynn Straus made changes -
            Link This issue blocks MB-36676 [ MB-36676 ]
            owend Daniel Owen made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            owend Daniel Owen made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Daniel Owen [ owend ]
            owend Daniel Owen made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Ben Huddleston [ ben.huddleston ]
            owend Daniel Owen made changes -
            Is this a Regression? Unknown [ 10452 ] No [ 10451 ]
            lynn.straus Lynn Straus made changes -
            Due Date 10/Dec/19 11/Dec/19
            drigby Dave Rigby made changes -
            Assignee Ben Huddleston [ ben.huddleston ] Richard deMellow [ richard.demellow ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            ritam.sharma Ritam Sharma made changes -
            Labels approved-for-mad-hatter approved-for-mad-hatter request-dev-verify
            richard.demellow Richard deMellow made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              richard.demellow Richard deMellow
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty