Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40076

Memcached crashed with EventuallyPersistentEngine::notifyIOComplete(void const*, ENGINE_ERROR_CODE)

    XMLWordPrintable

Details

    Description

      Steps to repro:

      1. Created a 7 node clusted
      2. Loaded 10 Million items of size 1024 Bytes(replicas =2)
      3. Remove a node and rebalance it
      4. During rebalance view queries were performed
      5. Observed Memcached crashed

      BackTrace:

      (gdb) bt full
      #0  0x000000000046c296 in notify_io_complete(gsl::not_null<void const*>, ENGINE_ERROR_CODE) ()
      No symbol table info available.
      #1  0x00007fb49a805788 in EventuallyPersistentEngine::notifyIOComplete(void const*, ENGINE_ERROR_CODE) () from /opt/couchbase/bin/../lib/libep.so
      No symbol table info available.
      #2  0x00007fb49a7913b1 in DcpConnMap::manageConnections() () from /opt/couchbase/bin/../lib/libep.so
      No symbol table info available.
      #3  0x00007fb49a764ad6 in ConnManager::run() () from /opt/couchbase/bin/../lib/libep.so
      No symbol table info available.
      #4  0x00007fb49a83c593 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
      No symbol table info available.
      #5  0x00007fb49a832daf in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
      No symbol table info available.
      #6  0x00007fb499240777 in run (this=0x7fb495e4c1f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58
      No locals.
      #7  platform_thread_wrap (arg=0x7fb495e4c1f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71
              context = {_M_t = {
                  _M_t = {<std::_Tuple_impl<0, CouchbaseThread*, std::default_delete<CouchbaseThread> >> = {<std::_Tuple_impl<1, std::default_delete<CouchbaseThread> >> = {<std::_Head_base<1, std::default_delete<CouchbaseThread>, true>> = {<std::default_delete<CouchbaseThread>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, CouchbaseThread*, false>> = {
                        _M_head_impl = 0x7fb495e4c1f0}, <No data fields>}, <No data fields>}}}
      #8  0x00007fb49687de65 in start_thread (arg=0x7fb4547f0700) at pthread_create.c:307
              __res = <optimized out>
              pd = 0x7fb4547f0700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140412488451840, -6877298850780401545, 0, 8392704, 0, 140412488451840, 6910287494748057719,
                      6910703458187358327}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0,
                    canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
      #9  0x00007fb4965a688d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      No locals.
      

      QE test:

      /testrunner -i /tmp/testexec.29978.ini num_items=10000000,GROUP=P1,bucket_storage=magma,bucket_eviction_policy=fullEviction,randomize_value=True,vbuckets=128,rerun=false,get-cbcollect-info=True,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-2416 -t rebalance_new.rebalance_out.RebalanceOutTests.rebalance_out_with_queries,nodes_init=7,nodes_out=1,blob_generator=False,value_size=1024,num_items=300000,dgm_run=True,max_verify=10000,replicas=2,GROUP=P1;not_for_ephemeral
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ankush.sharma Ankush Sharma
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty