Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39326

[Jepsen] Linearizability failure and crash in set-kill ephemeral test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • couchbase-bucket
    • Ubuntu 1804
      Couchbase Server 7.0.0-2047
    • Untriaged
    • 1
    • Unknown

    Description

      Linearizability failure and crash during kv-engine-cheshire-cat-jepsen-nightly-139 for test lein trampoline run test --nodes-file ./nodes --username root --password couchbase --workload=set-kill --scenario=kill-memcached --replicas=1 --no-autofailover --durability=0:100:0:0 --bucket-type=ephemeral --use-json-docs --use-subdoc --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --net-interface=eth0.

      (gdb) bt
      #0  Connection::remapErrorCode (this=0x0, code=code@entry=ENGINE_SUCCESS) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:283
      #1  0x00000000004cf270 in SteppableCommandContext::drive (this=this@entry=0x7febead46c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:54
      #2  0x00000000004a890d in add_set_replace_executor (store_op=OPERATION_ADD, cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:163
      #3  add_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:167
      #4  0x00000000004a9eb6 in std::function<void (Cookie&)>::operator()(Cookie&) const (__args#0=..., this=<optimized out>) at /usr/local/include/c++/7.3.0/bits/std_function.h:706
      #5  execute_client_request_packet (cookie=..., request=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:897
      #6  0x0000000000498f89 in Cookie::execute (this=0x7febeae61800) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:153
      #7  0x000000000047f269 in Connection::processAllReadyCookies (this=this@entry=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:501
      #8  0x0000000000481ab7 in Connection::executeCommandPipeline (this=this@entry=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:528
      #9  0x0000000000485258 in Connection::executeCommandsCallback (this=this@entry=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:638
      #10 0x0000000000485995 in Connection::rw_callback (ctx=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:762
      #11 0x00007febf04b871e in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7febe0019960) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208
      #12 0x00007febf04c1691 in event_process_active_single_queue (base=base@entry=0x2429380, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726
      #13 0x00007febf04c1edf in event_process_active (base=0x2429380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789
      #14 event_base_loop (base=0x2429380, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012
      #15 0x000000000046a0a9 in worker_libevent (arg=0x7febedfddaa8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:210
      #16 0x00007febf13f2437 in CouchbaseThread::run (this=0x7febede3cd70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
      #17 platform_thread_wrap (arg=0x7febede3cd70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
      #18 0x00007febee9816db in start_thread (arg=0x7febe9c78700) at pthread_create.c:463
      #19 0x00007febee6aa88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
      

      (gdb) bt full
      #0  Connection::remapErrorCode (this=0x0, code=code@entry=ENGINE_SUCCESS) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:283
              errc = <optimized out>
      #1  0x00000000004cf270 in SteppableCommandContext::drive (this=this@entry=0x7febead46c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:54
              ret = ENGINE_SUCCESS
              remapErr = <optimized out>
      #2  0x00000000004a890d in add_set_replace_executor (store_op=OPERATION_ADD, cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:163
      No locals.
      #3  add_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:167
      No locals.
      #4  0x00000000004a9eb6 in std::function<void (Cookie&)>::operator()(Cookie&) const (__args#0=..., this=<optimized out>) at /usr/local/include/c++/7.3.0/bits/std_function.h:706
      No locals.
      #5  execute_client_request_packet (cookie=..., request=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:897
              c = <optimized out>
              privilegeChains = {commandChains = {_M_elems = {{chain = {<std::_Vector_base<Function<cb::rbac::PrivilegeAccess, cb::rbac::PrivilegeAccess::getSuccessValue, Cookie&>, std::allocator<Function<cb::rbac::PrivilegeAccess, cb::rbac::PrivilegeAccess::getSuccessValue, Cookie&> > >> = {
                          _M_impl = {<std::allocator<Function<cb::rbac::PrivilegeAccess, cb::rbac::PrivilegeAccess::getSuccessValue, Cookie&> >> = {<__gnu_cxx::new_allocator<Function<cb::rbac::PrivilegeAccess, cb::rbac::PrivilegeAccess::getSuccessValue, Cookie&> >> = {<No data fields>}, <No data fields>}, _M_start = 0x0, _M_finish = 0x0, 
                            _M_end_of_storage = 0x0}}, <No data fields>}} <repeats 256 times>}}}
              opcode = <optimized out>
              res = {status = cb::rbac::PrivilegeAccess::Status::Ok}
      #6  0x0000000000498f89 in Cookie::execute (this=0x7febeae61800) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:153
              header = <error reading variable>
      #7  0x000000000047f269 in Connection::processAllReadyCookies (this=this@entry=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:501
              cookie = <optimized out>
              active = false
              iter = {_M_cur = 0x7febeace30a8, _M_first = <optimized out>, _M_last = 0x7febeace3200, _M_node = 0x7febede39da8}
      #8  0x0000000000481ab7 in Connection::executeCommandPipeline (this=this@entry=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:528
              maxActiveCommands = 32
              active = <optimized out>
              maxSendQueueSize = <optimized out>
      #9  0x0000000000485258 in Connection::executeCommandsCallback (this=this@entry=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:638
              start = <optimized out>
              stop = <optimized out>
              ns = <optimized out>
      #10 0x0000000000485995 in Connection::rw_callback (ctx=0x7febedf87200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:762
              instance = <error reading variable>
              thread = <optimized out>
              phosphor_internal_category_enabled_741 = {_M_b = {Segmentation fault (core dumped)
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              richard.demellow Richard deMellow
              richard.demellow Richard deMellow
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty