Basic crash dump analysis of /tmp/core.memcached.21045.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 396049 Oct 29 18:26 /opt/couchbase/bin/memcached
a7af118943eb2f6bf525ac17a3ff1a7d  /opt/couchbase/bin/memcached
-rw------- 1 couchbase couchbase 5619232768 Oct 31 07:24 /tmp/core.memcached.21045
fe746dca060e59a52c7c7092a14eb5b5  /tmp/core.memcached.21045
ep_version1.7.1.1_192_g4bebd09
ep_version1.7.1.1_192_g4bebd09
ep_version1.7.1.1_192_g4bebd09
1.7.1.1_192_g4bebd09
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) Amazon Linux (7.2-48.10.amzn1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 22692]
[New Thread 22695]
[New Thread 22693]
[New Thread 22694]
[New Thread 21052]
[New Thread 21053]
[New Thread 21054]
[New Thread 21057]
[New Thread 21045]
[New Thread 22680]
[New Thread 22691]
[New Thread 21051]
[New Thread 22690]
[New Thread 21056]
[New Thread 21055]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f8e0ee57700 (LWP 21055)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xb44c80, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xb44c80, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0xb26700) at daemon/thread.c:304
#4  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f8e0e656700 (LWP 21056)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xb44a00, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xb44a00, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0xb26800) at daemon/thread.c:304
#4  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f8e0d17f700 (LWP 22690)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xe6298000, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xe6298000, flags=0) at event.c:1558
#3  0x00007f8e0d33736e in MemcachedEngine::run (this=0xc1d000) at mc-kvstore/mc-engine.cc:590
#4  0x00007f8e0d3373b3 in start_memcached_engine (arg=0xc1d000) at mc-kvstore/mc-engine.cc:40
#5  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f8e11064700 (LWP 21051)):
#0  0x00007f8e11856b5d in read () from /lib64/libc.so.6
#1  0x00007f8e117f4228 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x00007f8e117f5d2e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3  0x00007f8e117f124b in getc () from /lib64/libc.so.6
#4  0x00007f8e11065875 in check_stdin_thread (arg=0x403610) at extensions/daemon/stdin_check.c:19
#5  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f8e0af6e700 (LWP 22691)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xe6298500, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xe6298500, flags=0) at event.c:1558
#3  0x00007f8e0d33736e in MemcachedEngine::run (this=0xc1f800) at mc-kvstore/mc-engine.cc:590
#4  0x00007f8e0d3373b3 in start_memcached_engine (arg=0xc1f800) at mc-kvstore/mc-engine.cc:40
#5  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f8e09f6c700 (LWP 22680)):
#0  0x00007f8e11b007a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f8e0fe5d7e7 in engine_shutdown_thread (arg=0xb490c0) at bucket_engine.c:1432
#2  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f8e12c43720 (LWP 21045)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xb44000, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xb44000, flags=0) at event.c:1558
#3  0x000000000040930b in main (argc=15, argv=0x7fffc1bd3758) at daemon/memcached.c:7531

Thread 8 (Thread 0x7f8e0de55700 (LWP 21057)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xb44780, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xb44780, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0xb26900) at daemon/thread.c:304
#4  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f8e0f658700 (LWP 21054)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xb44280, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xb44280, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0xb26600) at daemon/thread.c:304
#4  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f8e0fe59700 (LWP 21053)):
#0  0x00007f8e118644a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f8e1260ec28 in epoll_dispatch (base=0xb44500, tv=<value optimized out>) at epoll.c:404
#2  0x00007f8e125fda4c in event_base_loop (base=0xb44500, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0xb26500) at daemon/thread.c:304
#4  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f8e10863700 (LWP 21052)):
#0  0x00007f8e1182946d in nanosleep () from /lib64/libc.so.6
#1  0x00007f8e118292e0 in sleep () from /lib64/libc.so.6
#2  0x0000000000415068 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
#3  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f8e0b97c700 (LWP 22694)):
#0  0x00007f8e11b007a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f8e0d296bfc in SyncObject::wait (this=0xc69e10, tv=...) at syncobject.hh:42
#2  0x00007f8e0d293137 in IdleTask::run (this=0xb58500, d=...) at dispatcher.cc:244
#3  0x00007f8e0d29423c in Dispatcher::run (this=0xc69e00) at dispatcher.cc:119
#4  0x00007f8e0d29479c in launch_dispatcher_thread (arg=0xc69e00) at dispatcher.cc:28
#5  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f8e0c17d700 (LWP 22693)):
#0  0x00007f8e11b0043c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f8e0d297bb6 in SyncObject::wait (this=0x7f8e0c17cae0) at syncobject.hh:31
#2  0x00007f8e0d33e9ac in RememberingCallback<bool>::waitForValue (this=0x7f8e0c17cad0) at ./callbacks.hh:140
#3  0x00007f8e0d33d628 in MCKVStore::snapshotVBuckets (this=0x1411f0e10, m=...) at mc-kvstore/mc-kvstore.cc:107
#4  0x00007f8e0d29e499 in EventuallyPersistentStore::snapshotVBuckets (this=0x12f9b26c0, priority=...) at ep.cc:999
#5  0x00007f8e0d2ba9ef in SnapshotVBucketsCallback::callback (this=0xdc68eb60) at ep.cc:197
#6  0x00007f8e0d29568e in Task::run (this=0xc4a5ec80, d=..., t=...) at dispatcher.hh:139
#7  0x00007f8e0d29423c in Dispatcher::run (this=0xf5b91c0) at dispatcher.cc:119
#8  0x00007f8e0d29479c in launch_dispatcher_thread (arg=0xf5b91c0) at dispatcher.cc:28
#9  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f8e0a76d700 (LWP 22695)):
#0  0x00007f8e11b007a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f8e0d296bfc in SyncObject::wait (this=0xb46100, tv=...) at syncobject.hh:42
#2  0x00007f8e0d2e093e in SyncObject::wait (this=0xb46100, secs=1) at syncobject.hh:58
#3  0x00007f8e0d2e0979 in TapConnMap::wait (this=0xb46100, howlong=1) at tapconnmap.hh:209
#4  0x00007f8e0d2c391f in EventuallyPersistentEngine::notifyTapIoThread (this=0xd78000) at ep_engine.cc:3391
#5  0x00007f8e0d2ccaf4 in EvpNotifyTapIo (arg=0xd78000) at ep_engine.cc:1035
#6  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f8e11863ead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f8e0c97e700 (LWP 22692)):
#0  0x0000000000000000 in ?? ()
#1  0x00007f8e0d294b1c in Task::maxExpectedDuration (this=0x4b2c3000) at dispatcher.hh:152
#2  0x00007f8e0d2945a2 in Dispatcher::run (this=0xf5b9000) at dispatcher.cc:136
#3  0x00007f8e0d29479c in launch_dispatcher_thread (arg=0xf5b9000) at dispatcher.cc:28
#4  0x00007f8e11afc7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f8e11863ead in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Oct 29 23:04 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
13c89aae52d83681cef2a67531dbbc65  /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Oct 29 23:04 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
3a4104b298d23d059d6b2fbc5746ef98  /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 13 Feb 24  2011 /lib64/libdl.so.2 -> libdl-2.12.so
643aa33f4e56d9dc502a3915703335a0  /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 12 Feb 24  2011 /lib64/libm.so.6 -> libm-2.12.so
b496148ca454ca96f9e39c88730b2f7a  /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 13 Feb 24  2011 /lib64/librt.so.1 -> librt-2.12.so
d399fe4ea992cffb7227bbbc5dd827e6  /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.0:
lrwxrwxrwx 1 bin bin 28 Oct 29 23:04 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0
4c88156c56aabbd2a6474b31a6fe21ef  /opt/couchbase/lib/libtcmalloc_minimal.so.0
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 18 Feb 24  2011 /lib64/libpthread.so.0 -> libpthread-2.12.so
e3cc21f2a81e11035cfadb97580867ef  /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 12 Feb 24  2011 /lib64/libc.so.6 -> libc-2.12.so
b3e39b32eddb15662ba7d1c700a65bf0  /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 10 Feb 24  2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so
bdfe53856f4614f5fb1a4fcad54d235c  /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 Feb 24  2011 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13
0eb10e4f59017e1796fc0b1e227b9467  /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Feb 24  2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1
704445fcf4f7f80bf04c446957c31862  /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 15410 Oct 29 18:26 /opt/couchbase/lib/memcached/stdin_term_handler.so
61e9fec1f8429c34a8a105df0c928631  /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Oct 29 23:04 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
9f536ac339f17a605f874c4739c79fae  /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Oct 29 23:04 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
2502138c53d3467522493d077e77740b  /opt/couchbase/lib/memcached/ep.so
1.7.1.1_192_g4bebd09
CEP engine v1.7.1.1_192_g4bebd09
/lib64/libnss_files.so.2:
lrwxrwxrwx 1 root root 20 Feb 24  2011 /lib64/libnss_files.so.2 -> libnss_files-2.12.so
f4236af9b9a52a21f4d5e3fb7b865590  /lib64/libnss_files.so.2