Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.5.0
-
None
-
CentOS 7.4
-
Untriaged
-
Centos 64-bit
-
Yes
Description
memcached keeps crashing upon initialization if the number of previously available CPU cores has decreased.
Steps:
- Set up a new cluster
- Disable some of the cores via "echo 0 > /sys/devices/system/cpu/cpuX/online"
- Restart memcached
2018-01-28T18:15:05.878427Z CRITICAL *** Fatal error encountered during exception handling ***
|
2018-01-28T18:15:05.878470Z CRITICAL Caught unhandled std::exception-derived exception. what(): CoreStore::get index:26 out of bounds:12
|
2018-01-28T18:15:05.970636Z CRITICAL Breakpad caught crash in memcached version 5.5.0-1773. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/3d8dd121-9f54-b642-17c7a24b-1e894242.dmp before terminating.
|
2018-01-28T18:15:05.970665Z CRITICAL Stack backtrace of crashed thread:
|
2018-01-28T18:15:05.970790Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x38054]
|
2018-01-28T18:15:05.970807Z CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0xa070e]
|
2018-01-28T18:15:05.970823Z CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0xa0a24]
|
2018-01-28T18:15:05.970843Z CRITICAL /lib64/libpthread.so.0() [0x7ff5ede73000+0xf5e0]
|
2018-01-28T18:15:05.970879Z CRITICAL /lib64/libc.so.6(gsignal+0x37) [0x7ff5ebe29000+0x351f7]
|
2018-01-28T18:15:05.970913Z CRITICAL /lib64/libc.so.6(abort+0x148) [0x7ff5ebe29000+0x368e8]
|
2018-01-28T18:15:05.970950Z CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7ff5ec705000+0x911b5]
|
2018-01-28T18:15:05.970961Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x3945a]
|
2018-01-28T18:15:05.970993Z CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5ec705000+0x8efa6]
|
2018-01-28T18:15:05.971027Z CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5ec705000+0x8eff1]
|
2018-01-28T18:15:05.971055Z CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7ff5ec705000+0x8f233]
|
2018-01-28T18:15:05.971065Z CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so(create_instance+0x2d2) [0x7ff5d7dfd000+0x94992]
|
2018-01-28T18:15:05.971070Z CRITICAL /opt/couchbase/bin/../lib/libmcd_util.so.1.0.0(create_engine_instance+0x24) [0x7ff5efd6a000+0x4864]
|
2018-01-28T18:15:05.971082Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x4fc75]
|
2018-01-28T18:15:05.971091Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1aec9]
|
2018-01-28T18:15:05.971098Z CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1b379]
|
2018-01-28T18:15:05.971106Z CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0(_ZN9Couchbase6Thread12thread_entryEv+0xf) [0x7ff5ee4b3000+0x109bf]
|
2018-01-28T18:15:05.971114Z CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7ff5ee4b3000+0x8727]
|
2018-01-28T18:15:05.971123Z CRITICAL /lib64/libpthread.so.0() [0x7ff5ede73000+0x7e25]
|
2018-01-28T18:15:05.971175Z CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7ff5ebe29000+0xf834d]
|
(gdb) bt full
|
#0 0x00007fb8a79981f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
|
resultvar = 0
|
pid = 33467
|
selftid = 33489
|
#1 0x00007fb8a79998e8 in __GI_abort () at abort.c:90
|
save_stage = 2
|
act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 11 times>, 10, 140430062652512, 140430482074416, 140431062837235, 140431066268096}},
|
sa_flags = -1473457760, sa_restorer = 0x7fb8876c7798}
|
sigs = {__val = {32, 0 <repeats 15 times>}}
|
#2 0x00007fb8a82d01b5 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/lib/libstdc++.so
|
No symbol table info available.
|
#3 0x000000000043945a in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:103
|
No locals.
|
#4 0x00007fb8a82cdfa6 in ?? () from /opt/couchbase/lib/libstdc++.so
|
No symbol table info available.
|
#5 0x00007fb8a82cdff1 in std::terminate() () from /opt/couchbase/lib/libstdc++.so
|
No symbol table info available.
|
#6 0x00007fb8a82ce233 in __cxa_throw () from /opt/couchbase/lib/libstdc++.so
|
No symbol table info available.
|
#7 0x00007fb893a91992 in get (this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/corestore.h:50
|
index = <optimized out>
|
#8 create_instance (interface=<optimized out>, get_server_api=<optimized out>, handle=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:1631
|
api = <optimized out>
|
inital_tracking = <optimized out>
|
engine = <optimized out>
|
#9 0x00007fb8ab8a8864 in create_engine_instance (engine_ref=<optimized out>, get_server_api=get_server_api@entry=0x417180 <get_server_api()>, logger=0x7fb8ab8a2520 <descriptor>,
|
engine_handle=engine_handle@entry=0x70fd28 <all_buckets+185768>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/engine_loader.cc:125
|
engine = 0x0
|
error = <optimized out>
|
#10 0x000000000044fc75 in createInstance (this=<optimized out>, this=<optimized out>, handle=0x70fd28 <all_buckets+185768>, get_server_api=0x417180 <get_server_api()>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/enginemap.cc:45
|
No locals.
|
#11 new_engine_instance(enum BucketType, GET_SERVER_API, ENGINE_HANDLE **, EXTENSION_LOGGER_DESCRIPTOR *) (type=<optimized out>, get_server_api=get_server_api@entry=0x417180 <get_server_api()>,
|
handle=handle@entry=0x70fd28 <all_buckets+185768>, logger=0x7fb8ab8a2520 <descriptor>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/enginemap.cc:85
|
iter = <optimized out>
|
ret = <optimized out>
|
#12 0x000000000041aec9 in CreateBucketThread::create (this=this@entry=0x7fb8a5f2fdd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1985
|
ii = <optimized out>
|
first_free = <optimized out>
|
found = false
|
all_bucket_lock = {_M_device = 0x18bf840 <buckets_lock>, _M_owns = false}
|
#13 0x000000000041b379 in CreateBucketThread::run (this=0x7fb8a5f2fdd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:2054
|
guard = <optimized out>
|
#14 0x00007fb8a9ffd9bf in Couchbase::Thread::thread_entry (this=0x7fb8a5f2fdd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45
|
No locals.
|
#15 0x00007fb8a9ff5727 in run (this=0x7fb88faf5980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:59
|
No locals.
|
#16 platform_thread_wrap (arg=0x7fb88faf5980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:72
|
context = {_M_t = {
|
_M_t = {<std::_Tuple_impl<0, CouchbaseThread*, std::default_delete<CouchbaseThread> >> = {<std::_Tuple_impl<1, std::default_delete<CouchbaseThread> >> = {<std::_Head_base<1, std::default_delete<CouchbaseThread>, true>> = {<std::default_delete<CouchbaseThread>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, CouchbaseThread*, false>> = {
|
_M_head_impl = 0x7fb88faf5980}, <No data fields>}, <No data fields>}}}
|
#17 0x00007fb8a99b4e25 in start_thread (arg=0x7fb884fff700) at pthread_create.c:308
|
__res = <optimized out>
|
pd = 0x7fb884fff700
|
now = <optimized out>
|
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140430482077440, 7473018419703054624, 0, 140430482078144, 140430482077440, 140431034875344, -7438689467216821984, -7438770527184879328},
|
mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
|
not_first_call = <optimized out>
|
pagesize_m1 = <optimized out>
|
sp = <optimized out>
|
freesize = <optimized out>
|
#18 0x00007fb8a7a5b34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
|
|
Attachments
Issue Links
- is caused by
-
MB-24860 Leak on shutdown triggered by thread-local counters
- Closed