Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27767

memcached crashes due to out of bounds errors

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.5.0
    • 5.5.0
    • couchbase-bucket
    • None
    • CentOS 7.4
    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      memcached keeps crashing upon initialization if the number of previously available CPU cores has decreased.

      Steps:

      • Set up a new cluster
      • Disable some of the cores via "echo 0 > /sys/devices/system/cpu/cpuX/online"
      • Restart memcached

      2018-01-28T18:15:05.878427Z CRITICAL *** Fatal error encountered during exception handling ***
      2018-01-28T18:15:05.878470Z CRITICAL Caught unhandled std::exception-derived exception. what(): CoreStore::get index:26 out of bounds:12
      2018-01-28T18:15:05.970636Z CRITICAL Breakpad caught crash in memcached version 5.5.0-1773. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3d8dd121-9f54-b642-17c7a24b-1e894242.dmp before terminating.
      2018-01-28T18:15:05.970665Z CRITICAL Stack backtrace of crashed thread:
      2018-01-28T18:15:05.970790Z CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x38054]
      2018-01-28T18:15:05.970807Z CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0xa070e]
      2018-01-28T18:15:05.970823Z CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0xa0a24]
      2018-01-28T18:15:05.970843Z CRITICAL     /lib64/libpthread.so.0() [0x7ff5ede73000+0xf5e0]
      2018-01-28T18:15:05.970879Z CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7ff5ebe29000+0x351f7]
      2018-01-28T18:15:05.970913Z CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7ff5ebe29000+0x368e8]
      2018-01-28T18:15:05.970950Z CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7ff5ec705000+0x911b5]
      2018-01-28T18:15:05.970961Z CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x3945a]
      2018-01-28T18:15:05.970993Z CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5ec705000+0x8efa6]
      2018-01-28T18:15:05.971027Z CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5ec705000+0x8eff1]
      2018-01-28T18:15:05.971055Z CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5ec705000+0x8f233]
      2018-01-28T18:15:05.971065Z CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so(create_instance+0x2d2) [0x7ff5d7dfd000+0x94992]
      2018-01-28T18:15:05.971070Z CRITICAL     /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0(create_engine_instance+0x24) [0x7ff5efd6a000+0x4864]
      2018-01-28T18:15:05.971082Z CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x4fc75]
      2018-01-28T18:15:05.971091Z CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1aec9]
      2018-01-28T18:15:05.971098Z CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1b379]
      2018-01-28T18:15:05.971106Z CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0xf) [0x7ff5ee4b3000+0x109bf]
      2018-01-28T18:15:05.971114Z CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7ff5ee4b3000+0x8727]
      2018-01-28T18:15:05.971123Z CRITICAL     /lib64/libpthread.so.0() [0x7ff5ede73000+0x7e25]
      2018-01-28T18:15:05.971175Z CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7ff5ebe29000+0xf834d]
      

      (gdb) bt full
      #0  0x00007fb8a79981f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
              resultvar = 0
              pid = 33467
              selftid = 33489
      #1  0x00007fb8a79998e8 in __GI_abort () at abort.c:90
              save_stage = 2
              act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 11 times>, 10, 140430062652512, 140430482074416, 140431062837235, 140431066268096}}, 
                sa_flags = -1473457760, sa_restorer = 0x7fb8876c7798}
              sigs = {__val = {32, 0 <repeats 15 times>}}
      #2  0x00007fb8a82d01b5 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/lib/libstdc++.so
      No symbol table info available.
      #3  0x000000000043945a in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:103
      No locals.
      #4  0x00007fb8a82cdfa6 in ?? () from /opt/couchbase/lib/libstdc++.so
      No symbol table info available.
      #5  0x00007fb8a82cdff1 in std::terminate() () from /opt/couchbase/lib/libstdc++.so
      No symbol table info available.
      #6  0x00007fb8a82ce233 in __cxa_throw () from /opt/couchbase/lib/libstdc++.so
      No symbol table info available.
      #7  0x00007fb893a91992 in get (this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/corestore.h:50
              index = <optimized out>
      #8  create_instance (interface=<optimized out>, get_server_api=<optimized out>, handle=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:1631
              api = <optimized out>
              inital_tracking = <optimized out>
              engine = <optimized out>
      #9  0x00007fb8ab8a8864 in create_engine_instance (engine_ref=<optimized out>, get_server_api=get_server_api@entry=0x417180 <get_server_api()>, logger=0x7fb8ab8a2520 <descriptor>, 
          engine_handle=engine_handle@entry=0x70fd28 <all_buckets+185768>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/engine_loader.cc:125
              engine = 0x0
              error = <optimized out>
      #10 0x000000000044fc75 in createInstance (this=<optimized out>, this=<optimized out>, handle=0x70fd28 <all_buckets+185768>, get_server_api=0x417180 <get_server_api()>)
          at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/enginemap.cc:45
      No locals.
      #11 new_engine_instance(enum BucketType, GET_SERVER_API, ENGINE_HANDLE **, EXTENSION_LOGGER_DESCRIPTOR *) (type=<optimized out>, get_server_api=get_server_api@entry=0x417180 <get_server_api()>, 
          handle=handle@entry=0x70fd28 <all_buckets+185768>, logger=0x7fb8ab8a2520 <descriptor>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/enginemap.cc:85
              iter = <optimized out>
              ret = <optimized out>
      #12 0x000000000041aec9 in CreateBucketThread::create (this=this@entry=0x7fb8a5f2fdd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1985
              ii = <optimized out>
              first_free = <optimized out>
              found = false
              all_bucket_lock = {_M_device = 0x18bf840 <buckets_lock>, _M_owns = false}
      #13 0x000000000041b379 in CreateBucketThread::run (this=0x7fb8a5f2fdd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:2054
              guard = <optimized out>
      #14 0x00007fb8a9ffd9bf in Couchbase::Thread::thread_entry (this=0x7fb8a5f2fdd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45
      No locals.
      #15 0x00007fb8a9ff5727 in run (this=0x7fb88faf5980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:59
      No locals.
      #16 platform_thread_wrap (arg=0x7fb88faf5980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:72
              context = {_M_t = {
                  _M_t = {<std::_Tuple_impl<0, CouchbaseThread*, std::default_delete<CouchbaseThread> >> = {<std::_Tuple_impl<1, std::default_delete<CouchbaseThread> >> = {<std::_Head_base<1, std::default_delete<CouchbaseThread>, true>> = {<std::default_delete<CouchbaseThread>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, CouchbaseThread*, false>> = {
                        _M_head_impl = 0x7fb88faf5980}, <No data fields>}, <No data fields>}}}
      #17 0x00007fb8a99b4e25 in start_thread (arg=0x7fb884fff700) at pthread_create.c:308
              __res = <optimized out>
              pd = 0x7fb884fff700
              now = <optimized out>
              unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140430482077440, 7473018419703054624, 0, 140430482078144, 140430482077440, 140431034875344, -7438689467216821984, -7438770527184879328}, 
                    mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
              not_first_call = <optimized out>
              pagesize_m1 = <optimized out>
              sp = <optimized out>
              freesize = <optimized out>
      #18 0x00007fb8a7a5b34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
      
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty