Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39435

[Jepsen] Multiple crashes during Checkpoint::expelItems() due to Expects() throwing

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.6.0
    • 6.6.0
    • couchbase-bucket
    • Ubuntu 1804

    Description

      Crash in memcached seen during kv-engine-madhatter-jepsen-post-commit/434 during test lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --install-path /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install --workload=kill --scenario=kill-babysitter --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --cas --use-json-docs --use-subdoc --doc-padding-size=4096 --hashdump --enable-memcached-debug-log-level --enable-tcp-capture --autofailover-timeout=30 on node 172.28.128.196. For patch 2bd86cd710b5f8f07feb908b29b3f4a48b383b8c.

      2020-05-18T08:07:35.545499+00:00 CRITICAL *** Fatal error encountered during exception handling ***
      2020-05-18T08:07:35.546653+00:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/kv_engine/engines/ep/src/checkpoint.cc: 470
      2020-05-18T08:07:35.844671+00:00 CRITICAL Breakpad caught a crash (Couchbase version 9.9.9-9999). Writing crash dump to /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/var/lib/couchbase/crash/5669ad2e-97c5-e7e4-67bfb236-1b43691d.dmp before terminating.
      2020-05-18T08:07:35.844728+00:00 CRITICAL Stack backtrace of crashed thread:
      2020-05-18T08:07:35.845302+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/memcached() [0x400000+0x13db6d]
      2020-05-18T08:07:35.845343+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x1570ce]
      2020-05-18T08:07:35.845361+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x1573e4]
      2020-05-18T08:07:35.845382+00:00 CRITICAL     /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f733ce84000+0x11390]
      2020-05-18T08:07:35.845603+00:00 CRITICAL     /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f733caba000+0x35428]
      2020-05-18T08:07:35.845661+00:00 CRITICAL     /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f733caba000+0x3702a]
      2020-05-18T08:07:35.846111+00:00 CRITICAL     /usr/lib/x86_64-linux-gnu/libstdc++.so.6() [0x7f733d5c0000+0x99dde]
      2020-05-18T08:07:35.846146+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/memcached() [0x400000+0x152552]
      2020-05-18T08:07:35.846310+00:00 CRITICAL     /usr/lib/x86_64-linux-gnu/libstdc++.so.6() [0x7f733d5c0000+0xa5896]
      2020-05-18T08:07:35.846410+00:00 CRITICAL     /usr/lib/x86_64-linux-gnu/libstdc++.so.6() [0x7f733d5c0000+0xa5901]
      2020-05-18T08:07:35.846583+00:00 CRITICAL     /usr/lib/x86_64-linux-gnu/libstdc++.so.6() [0x7f733d5c0000+0xa5b55]
      2020-05-18T08:07:35.846704+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/../lib/../lib/ep.so() [0x7f7337c8b000+0x50dd8]
      2020-05-18T08:07:35.846729+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/../lib/../lib/ep.so() [0x7f7337c8b000+0x589cd]
      2020-05-18T08:07:35.846736+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/../lib/../lib/ep.so() [0x7f7337c8b000+0x650a4]
      2020-05-18T08:07:35.846745+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/../lib/../lib/ep.so() [0x7f7337c8b000+0x137304]
      2020-05-18T08:07:35.846757+00:00 CRITICAL     /home/couchbase/jenkins/workspace/kv-engine-madhatter-jepsen-post-commit/install/bin/../lib/libplatform_so.so.0.1.0() [0x7f733f076000+0x8e67]
      2020-05-18T08:07:35.846766+00:00 CRITICAL     /lib/x86_64-linux-gnu/libpthread.so.0() [0x7f733ce84000+0x76ba]
      2020-05-18T08:07:35.846956+00:00 CRITICAL     /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f733caba000+0x10741d]
      

      Expects being triggered in expelItems()

      Expects(itr != keyIndex.end());
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            This looks like an issue with MB-35889 where I didn't account for an assertion that checks if the item exists in the key index when expelling. Given that MB-35889 has approval already and this is an issue with the solution, will fix under that MB and close this one.

            ben.huddleston Ben Huddleston added a comment - This looks like an issue with MB-35889 where I didn't account for an assertion that checks if the item exists in the key index when expelling. Given that MB-35889 has approval already and this is an issue with the solution, will fix under that MB and close this one.
            ben.huddleston Ben Huddleston added a comment - Resolved by MB-35889 - http://review.couchbase.org/c/kv_engine/+/128352 .

            People

              owend Daniel Owen
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty