Basic crash dump analysis of /tmp/core.memcached.1420. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 396049 Oct 29 18:26 /opt/couchbase/bin/memcached a7af118943eb2f6bf525ac17a3ff1a7d /opt/couchbase/bin/memcached -rw------- 1 couchbase couchbase 5662859264 Oct 31 07:22 /tmp/core.memcached.1420 a49b5b69c756e4e59da52cb77882fe83 /tmp/core.memcached.1420 ep_version1.7.1.1_192_g4bebd09 ep_version1.7.1.1_192_g4bebd09 ep_version1.7.1.1_192_g4bebd09 1.7.1.1_192_g4bebd09 ep_version1.7.1.1_192_g4bebd09 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Amazon Linux (7.2-48.10.amzn1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-amazon-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 24590] [New Thread 24593] [New Thread 24592] [New Thread 1423] [New Thread 24591] [New Thread 1424] [New Thread 1420] [New Thread 1425] [New Thread 1426] [New Thread 1427] [New Thread 1422] [New Thread 1428] [New Thread 22441] [New Thread 24563] [New Thread 24588] [New Thread 24589] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 0x0000000000000000 in ?? () Thread 16 (Thread 0x7f878ce0c700 (LWP 24589)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x220a280, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x220a280, flags=0) at event.c:1558 #3 0x00007f878e9d436e in MemcachedEngine::run (this=0x5aa1800) at mc-kvstore/mc-engine.cc:590 #4 0x00007f878e9d43b3 in start_memcached_engine (arg=0x5aa1800) at mc-kvstore/mc-engine.cc:40 #5 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f878e81c700 (LWP 24588)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x220a780, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x220a780, flags=0) at event.c:1558 #3 0x00007f878e9d436e in MemcachedEngine::run (this=0x2207800) at mc-kvstore/mc-engine.cc:590 #4 0x00007f878e9d43b3 in start_memcached_engine (arg=0x2207800) at mc-kvstore/mc-engine.cc:40 #5 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f878ae08700 (LWP 24563)): #0 0x00007f879319d7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f87914fa7e7 in engine_shutdown_thread (arg=0x221b9c0) at bucket_engine.c:1432 #2 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f878b609700 (LWP 22441)): #0 0x00007f879319d7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f87914fa7e7 in engine_shutdown_thread (arg=0x221b0c0) at bucket_engine.c:1432 #2 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f878f4f2700 (LWP 1428)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x2216780, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x2216780, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x21f8900) at daemon/thread.c:304 #4 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f8792701700 (LWP 1422)): #0 0x00007f8792ef3b5d in read () from /lib64/libc.so.6 #1 0x00007f8792e91228 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00007f8792e92d2e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00007f8792e8e24b in getc () from /lib64/libc.so.6 #4 0x00007f8792702875 in check_stdin_thread (arg=0x403610) at extensions/daemon/stdin_check.c:19 #5 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f878fcf3700 (LWP 1427)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x2216a00, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x2216a00, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x21f8800) at daemon/thread.c:304 #4 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f87904f4700 (LWP 1426)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x2216c80, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x2216c80, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x21f8700) at daemon/thread.c:304 #4 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f8790cf5700 (LWP 1425)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x2216280, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x2216280, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x21f8600) at daemon/thread.c:304 #4 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f87942e0720 (LWP 1420)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x2216000, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x2216000, flags=0) at event.c:1558 #3 0x000000000040930b in main (argc=15, argv=0x7fffa8879f58) at daemon/memcached.c:7531 Thread 6 (Thread 0x7f87914f6700 (LWP 1424)): #0 0x00007f8792f014a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f8793cabc28 in epoll_dispatch (base=0x2216500, tv=) at epoll.c:404 #2 0x00007f8793c9aa4c in event_base_loop (base=0x2216500, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x21f8500) at daemon/thread.c:304 #4 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f878e01b700 (LWP 24591)): #0 0x00007f879319d7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f878e933bfc in SyncObject::wait (this=0x7588010, tv=...) at syncobject.hh:42 #2 0x00007f878e930137 in IdleTask::run (this=0x222a700, d=...) at dispatcher.cc:244 #3 0x00007f878e93123c in Dispatcher::run (this=0x7588000) at dispatcher.cc:119 #4 0x00007f878e93179c in launch_dispatcher_thread (arg=0x7588000) at dispatcher.cc:28 #5 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f8791f00700 (LWP 1423)): #0 0x00007f8792ec646d in nanosleep () from /lib64/libc.so.6 #1 0x00007f8792ec62e0 in sleep () from /lib64/libc.so.6 #2 0x0000000000415068 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f878d81a700 (LWP 24592)): #0 0x00007f879319d7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f878e933bfc in SyncObject::wait (this=0x22efe10, tv=...) at syncobject.hh:42 #2 0x00007f878e930137 in IdleTask::run (this=0x222a080, d=...) at dispatcher.cc:244 #3 0x00007f878e93123c in Dispatcher::run (this=0x22efe00) at dispatcher.cc:119 #4 0x00007f878e93179c in launch_dispatcher_thread (arg=0x22efe00) at dispatcher.cc:28 #5 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f878be0a700 (LWP 24593)): #0 0x00007f878e929dea in std::_List_iterator >::operator++ (this=0x7f878be09b20) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_list.h:141 #1 0x00007f878e925178 in CheckpointManager::hasNext (this=0x16d85080, name=...) at checkpoint.cc:914 #2 0x00007f878e9a3368 in TapProducer::hasNextFromCheckpoints_UNLOCKED (this=0x2313c00) at tapconnection.cc:1493 #3 0x00007f878e9833b0 in TapProducer::hasQueuedItem_UNLOCKED (this=0x2313c00) at tapconnection.hh:792 #4 0x00007f878e98348c in TapProducer::empty_UNLOCKED (this=0x2313c00) at tapconnection.hh:797 #5 0x00007f878e9b57bb in TapProducer::idle_UNLOCKED (this=0x2313c00) at tapconnection.hh:802 #6 0x00007f878e9b5855 in TapProducer::idle (this=0x2313c00) at tapconnection.hh:807 #7 0x00007f878e9b8e4e in TapConnMap::notifyIOThreadMain (this=0x2218200) at tapconnmap.cc:493 #8 0x00007f878e9608f8 in EventuallyPersistentEngine::notifyTapIoThread (this=0x31315000) at ep_engine.cc:3385 #9 0x00007f878e969af4 in EvpNotifyTapIo (arg=0x31315000) at ep_engine.cc:1035 #10 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f8792f00ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f878c60b700 (LWP 24590)): #0 0x0000000000000000 in ?? () #1 0x00007f878e931b1c in Task::maxExpectedDuration (this=0x102a94480) at dispatcher.hh:152 #2 0x00007f878e9315a2 in Dispatcher::run (this=0x7588e00) at dispatcher.cc:136 #3 0x00007f878e93179c in launch_dispatcher_thread (arg=0x7588e00) at dispatcher.cc:28 #4 0x00007f87931997e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8792f00ead in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 Oct 29 23:05 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 13c89aae52d83681cef2a67531dbbc65 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 /opt/couchbase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 Oct 29 23:05 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 3a4104b298d23d059d6b2fbc5746ef98 /opt/couchbase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 13 Feb 24 2011 /lib64/libdl.so.2 -> libdl-2.12.so 643aa33f4e56d9dc502a3915703335a0 /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 12 Feb 24 2011 /lib64/libm.so.6 -> libm-2.12.so b496148ca454ca96f9e39c88730b2f7a /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 13 Feb 24 2011 /lib64/librt.so.1 -> librt-2.12.so d399fe4ea992cffb7227bbbc5dd827e6 /lib64/librt.so.1 /opt/couchbase/lib/libtcmalloc_minimal.so.0: lrwxrwxrwx 1 bin bin 28 Oct 29 23:05 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0 4c88156c56aabbd2a6474b31a6fe21ef /opt/couchbase/lib/libtcmalloc_minimal.so.0 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 18 Feb 24 2011 /lib64/libpthread.so.0 -> libpthread-2.12.so e3cc21f2a81e11035cfadb97580867ef /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 12 Feb 24 2011 /lib64/libc.so.6 -> libc-2.12.so b3e39b32eddb15662ba7d1c700a65bf0 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 10 Feb 24 2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so bdfe53856f4614f5fb1a4fcad54d235c /lib64/ld-linux-x86-64.so.2 /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 19 Feb 24 2011 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 0eb10e4f59017e1796fc0b1e227b9467 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Feb 24 2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1 704445fcf4f7f80bf04c446957c31862 /lib64/libgcc_s.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15410 Oct 29 18:26 /opt/couchbase/lib/memcached/stdin_term_handler.so 61e9fec1f8429c34a8a105df0c928631 /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 Oct 29 23:05 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 9f536ac339f17a605f874c4739c79fae /opt/couchbase/lib/memcached/bucket_engine.so /opt/couchbase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 Oct 29 23:05 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0 2502138c53d3467522493d077e77740b /opt/couchbase/lib/memcached/ep.so 1.7.1.1_192_g4bebd09 CEP engine v1.7.1.1_192_g4bebd09 /lib64/libnss_files.so.2: lrwxrwxrwx 1 root root 20 Feb 24 2011 /lib64/libnss_files.so.2 -> libnss_files-2.12.so f4236af9b9a52a21f4d5e3fb7b865590 /lib64/libnss_files.so.2