Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37204

[Jepsen] Couchbase crash during hard-reboot scenario due to HCS > persistedCompletedSeqno

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 6.5.0
    • None
    • couchbase-bucket
    • None
    • Triaged
    • Yes
    • KV Sprint 2019-12

    Description

      During kv-engine-jepsen-nightly-352 we observed a and crash during the following test:
      lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=kill --scenario=hard-reboot --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --cas
      Crash Message:

      2019-12-08T01:36:51.777389+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 620
      

      ep_bucket.cc:620

                      if (hcs) {
                          Expects(hcs > vbstate.persistedCompletedSeqno);
                          vbstate.persistedCompletedSeqno = *hcs;
                      }
      

      Couchbase version 6.5.1-6001

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            richard.demellow Richard deMellow created issue -
            richard.demellow Richard deMellow made changes -
            Field Original Value New Value
            Summary [Jepsen] Crash [Jepsen] Crash during hard-reboot scenario
            richard.demellow Richard deMellow made changes -
            Remote Link This issue links to "20191207T181623.000-0800.zip (Web Link)" [ 18944 ]
            richard.demellow Richard deMellow made changes -
            Summary [Jepsen] Crash during hard-reboot scenario [Jepsen] Crashes during hard-reboot scenario
            richard.demellow Richard deMellow made changes -
            Description During [kv-engine-jepsen-nightly-352|http://cv.jenkins.couchbase.com/view/Commit%20Validation/view/Jepsen/job/kv-engine-jepsen-nightly/352/] we observed a and crash during the following test:
            {{lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=kill --scenario=hard-reboot --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --cas}}
            Crash Message:
            {noformat}
            2019-12-08T01:36:51.777389+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 620
            {noformat}

            {code:cpp|title=ep_bucket.cc:620}
                            if (hcs) {
                                Expects(hcs > vbstate.persistedCompletedSeqno);
                                vbstate.persistedCompletedSeqno = *hcs;
                            }
            {code}
            During [kv-engine-jepsen-nightly-352|http://cv.jenkins.couchbase.com/view/Commit%20Validation/view/Jepsen/job/kv-engine-jepsen-nightly/352/] we observed a and crash during the following test:
            {{lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=kill --scenario=hard-reboot --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=full --cas}}
            Crash Message:
            {noformat}
            2019-12-08T01:36:51.777389+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc: 620
            {noformat}

            {code:cpp|title=ep_bucket.cc:620}
                            if (hcs) {
                                Expects(hcs > vbstate.persistedCompletedSeqno);
                                vbstate.persistedCompletedSeqno = *hcs;
                            }
            {code}
            Couchbase version 6.5.1-6001
            richard.demellow Richard deMellow made changes -
            Summary [Jepsen] Crashes during hard-reboot scenario [Jepsen] Couchbase crash during hard-reboot scenario due to HCS >
            richard.demellow Richard deMellow made changes -
            Summary [Jepsen] Couchbase crash during hard-reboot scenario due to HCS > [Jepsen] Couchbase crash during hard-reboot scenario due to HCS > persistedCompletedSeqno
            drigby Dave Rigby made changes -
            Assignee Daniel Owen [ owend ] Dave Rigby [ drigby ]
            owend Daniel Owen made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            owend Daniel Owen made changes -
            Fix Version/s Mad-Hatter [ 15037 ]
            Fix Version/s 6.5.0 [ 16624 ]
            owend Daniel Owen made changes -
            Affects Version/s 6.5.1 [ 16622 ]
            owend Daniel Owen made changes -
            Fix Version/s 6.5.1 [ 16622 ]
            drigby Dave Rigby made changes -
            Affects Version/s 6.5.1 [ 16622 ]
            drigby Dave Rigby made changes -
            Fix Version/s 6.5.1 [ 16622 ]
            Fix Version/s 6.5.0 [ 16624 ]
            owend Daniel Owen made changes -
            Sprint KV Sprint 2019-12 [ 939 ]
            owend Daniel Owen made changes -
            Rank Ranked higher
            drigby Dave Rigby made changes -
            Due Date 10/Dec/19
            drigby Dave Rigby made changes -
            Link This issue is caused by MB-37161 [ MB-37161 ]
            drigby Dave Rigby made changes -
            Is this a Regression? Unknown [ 10452 ] Yes [ 10450 ]
            drigby Dave Rigby made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            drigby Dave Rigby made changes -
            Fix Version/s Mad-Hatter [ 15037 ]
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            drigby Dave Rigby made changes -
            Link This issue duplicates MB-37161 [ MB-37161 ]
            drigby Dave Rigby made changes -
            Link This issue duplicates MB-37161 [ MB-37161 ]
            raju Raju Suravarjjala made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              drigby Dave Rigby
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty