Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37827

Race condition between bucket shutdown and seqno ack processing

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • KV Sprint 2020-February

    Description

      Crash found (on node 172.28.128.107)during set-kill workload during kv-engine-cheshire-cat-jepsen-nightly-29 for build 7.0.0-1283.

      Jepsen Test Command: lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=set-kill --scenario=kill-babysitter --replicas=2 --no-autofailover --durability=0:100:0:0 --eviction-policy=value --enable-tcp-capture --hashdump --enable-memcached-debug-log-level

      (gdb) bt
      #0  0x00007fcb50838428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
      #1  0x00007fcb5083a02a in __GI_abort () at abort.c:89
      #2  0x00007fcb51399d25 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
      #3  0x0000000000544462 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86
      #4  0x00007fcb51397b16 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
      #5  0x00007fcb51397b61 in std::terminate () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
      #6  0x00007fcb51397df6 in __cxxabiv1::__cxa_rethrow () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:131
      #7  0x0000000000544489 in log_handled_exception () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:44
      #8  backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:77
      #9  0x00007fcb51397b16 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
      #10 0x00007fcb51397b61 in std::terminate () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
      #11 0x00007fcb5139889f in __cxxabiv1::__cxa_pure_virtual () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/pure.cc:50
      #12 0x00007fcb550d7762 in ActiveStream::seqnoAck (this=this@entry=0x7fcb13e00c10, consumerName=..., preparedSeqno=308) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1876
      #13 0x00007fcb5511f0f4 in DcpProducer::seqno_acknowledged (this=0x7fcb13f6f910, opaque=6, vbucket=..., prepared_seqno=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:1069
      #14 0x00007fcb5516beeb in non-virtual thunk to EventuallyPersistentEngine::seqno_acknowledged(gsl::not_null<void const*>, unsigned int, Vbid, unsigned long) () at /usr/local/include/c++/7.3.0/bits/unique_ptr.h:825
      #15 0x0000000000452e75 in dcpSeqnoAcknowledged (cookie=..., opaque=6, vbucket=..., prepared_seqno=308) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:761
      #16 0x00000000004bbe6c in dcp_seqno_acknowledged_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/dcp_seqno_acknowledged_executor.cc:33
      #17 0x00000000004aa3a4 in std::function<void (Cookie&)>::operator()(Cookie&) const (__args#0=..., this=<optimized out>) at /usr/local/include/c++/7.3.0/bits/std_function.h:706
      #18 execute_client_request_packet (cookie=..., request=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:853
      #19 0x0000000000499b79 in Cookie::execute (this=this@entry=0x7fcb4cf7bc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:153
      #20 0x0000000000482a4b in Connection::executeCommandPipeline (this=this@entry=0x7fcb4cf77600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:568
      #21 0x0000000000485f28 in Connection::executeCommandsCallback (this=this@entry=0x7fcb4cf77600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:634
      #22 0x0000000000486655 in Connection::rw_callback (ctx=0x7fcb4cf77600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:757
      #23 0x00007fcb5286cd1e in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7fcb34001060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208
      #24 0x00007fcb52875c91 in event_process_active_single_queue (base=base@entry=0x202c4c0, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726
      #25 0x00007fcb528764df in event_process_active (base=0x202c4c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789
      #26 event_base_loop (base=0x202c4c0, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012
      #27 0x0000000000468db9 in worker_libevent (arg=0x7fcb501f9ff8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:210
      #28 0x00007fcb5359b867 in CouchbaseThread::run (this=0x7fcb4ceaf840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
      #29 platform_thread_wrap (arg=0x7fcb4ceaf840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
      #30 0x00007fcb50bd46ba in start_thread (arg=0x7fcb4ae76700) at pthread_create.c:333
      #31 0x00007fcb5090a41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sumedh.basarkod Sumedh Basarkod (Inactive)
            richard.demellow Richard deMellow
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty