Basic crash dump analysis of /tmp/core.memcached.2408.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 396049 Nov  6 01:09 /opt/couchbase/bin/memcached
a7af118943eb2f6bf525ac17a3ff1a7d  /opt/couchbase/bin/memcached
-rw------- 1 couchbase couchbase 163172352 Nov  7 22:22 /tmp/core.memcached.2408
37086c846162a5684225b28c606a2672  /tmp/core.memcached.2408
1.7.1.1_205_ge1a9877
ep_version1.7.1.1_205_ge1a9877
ep_version1.7.1.1_205_ge1a9877
ep_version1.7.1.1_205_ge1a9877
ep_version1.7.1.1_205_ge1a9877
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) Amazon Linux (7.2-48.10.amzn1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 2532]
[New Thread 2533]
[New Thread 2421]
[New Thread 2534]
[New Thread 2415]
[New Thread 2408]
[New Thread 2418]
[New Thread 2414]
[New Thread 2419]
[New Thread 2422]
[New Thread 2420]
[New Thread 2467]
[New Thread 2521]
[New Thread 2483]
[New Thread 2531]
[New Thread 2512]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0  0x00007fc19df3a6b5 in raise () from /lib64/libc.so.6

Thread 16 (Thread 0x7fc1946f0700 (LWP 2512)):
#0  0x00007fc19e2857a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc19c5e27e7 in engine_shutdown_thread (arg=0x6e8240) at bucket_engine.c:1432
#2  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fc1956f2700 (LWP 2531)):
#0  0x00007fc19e28543c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc199a1fe16 in SyncObject::wait (this=0x900390) at syncobject.hh:31
#2  0x00007fc199a1c1bd in Dispatcher::run (this=0x900380) at dispatcher.cc:85
#3  0x00007fc199a1c9fc in launch_dispatcher_thread (arg=0x900380) at dispatcher.cc:28
#4  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fc196ef5700 (LWP 2483)):
#0  0x00007fc19e2857a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc19c5e27e7 in engine_shutdown_thread (arg=0x6e8540) at bucket_engine.c:1432
#2  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fc1996fa700 (LWP 2521)):
#0  0x00007fc19e2857a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc19c5e27e7 in engine_shutdown_thread (arg=0x6e86c0) at bucket_engine.c:1432
#2  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fc1976f6700 (LWP 2467)):
#0  0x00007fc19e2857a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc19c5e27e7 in engine_shutdown_thread (arg=0x6e80c0) at bucket_engine.c:1432
#2  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fc19b5dc700 (LWP 2420)):
#0  0x00007fc19dfe94a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc19ed93c28 in epoll_dispatch (base=0x6e3c80, tv=<value optimized out>) at epoll.c:404
#2  0x00007fc19ed82a4c in event_base_loop (base=0x6e3c80, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x6c5700) at daemon/thread.c:304
#4  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fc19a5da700 (LWP 2422)):
#0  0x00007fc19dfe94a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc19ed93c28 in epoll_dispatch (base=0x6e3780, tv=<value optimized out>) at epoll.c:404
#2  0x00007fc19ed82a4c in event_base_loop (base=0x6e3780, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x6c5900) at daemon/thread.c:304
#4  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fc19bddd700 (LWP 2419)):
#0  0x00007fc19dfe94a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc19ed93c28 in epoll_dispatch (base=0x6e3280, tv=<value optimized out>) at epoll.c:404
#2  0x00007fc19ed82a4c in event_base_loop (base=0x6e3280, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x6c5600) at daemon/thread.c:304
#4  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fc19d7e9700 (LWP 2414)):
#0  0x00007fc19dfdbb5d in read () from /lib64/libc.so.6
#1  0x00007fc19df79228 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x00007fc19df7ad2e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3  0x00007fc19df7624b in getc () from /lib64/libc.so.6
#4  0x00007fc19d7ea875 in check_stdin_thread (arg=0x403610) at extensions/daemon/stdin_check.c:19
#5  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fc19c5de700 (LWP 2418)):
#0  0x00007fc19dfe94a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc19ed93c28 in epoll_dispatch (base=0x6e3500, tv=<value optimized out>) at epoll.c:404
#2  0x00007fc19ed82a4c in event_base_loop (base=0x6e3500, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x6c5500) at daemon/thread.c:304
#4  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fc19f3c8720 (LWP 2408)):
#0  0x00007fc19dfe94a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fc19ed93c28 in epoll_dispatch (base=0x6e3000, tv=<value optimized out>) at epoll.c:404
#2  0x00007fc19ed82a4c in event_base_loop (base=0x6e3000, flags=0) at event.c:1558
#3  0x000000000040930b in main (argc=15, argv=0x7ffffa545d98) at daemon/memcached.c:7531

Thread 5 (Thread 0x7fc19cfe8700 (LWP 2415)):
#0  0x00007fc19dfae46d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc19dfae2e0 in sleep () from /lib64/libc.so.6
#2  0x0000000000415068 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
#3  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fc194ef1700 (LWP 2534)):
#0  0x00007fc19e2857a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc199a1ee5c in SyncObject::wait (this=0x6e5100, tv=...) at syncobject.hh:42
#2  0x00007fc199a68a9c in SyncObject::wait (this=0x6e5100, secs=1) at syncobject.hh:58
#3  0x00007fc199a68ad7 in TapConnMap::wait (this=0x6e5100, howlong=1) at tapconnmap.hh:209
#4  0x00007fc199a4c04f in EventuallyPersistentEngine::notifyTapIoThread (this=0xa6a000) at ep_engine.cc:3393
#5  0x00007fc199a55224 in EvpNotifyTapIo (arg=0xa6a000) at ep_engine.cc:1037
#6  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc19addb700 (LWP 2421)):
#0  0x00007fc19e288afd in sendmsg () from /lib64/libpthread.so.0
#1  0x000000000040d92c in transmit (c=0x95a088) at daemon/memcached.c:5020
#2  conn_mwrite (c=0x95a088) at daemon/memcached.c:5419
#3  0x0000000000407556 in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x95a088) at daemon/memcached.c:5620
#4  0x00007fc19ed82df9 in event_process_active_single_queue (base=0x6e3a00, flags=0) at event.c:1308
#5  event_process_active (base=0x6e3a00, flags=0) at event.c:1375
#6  event_base_loop (base=0x6e3a00, flags=0) at event.c:1572
#7  0x00000000004134a4 in worker_libevent (arg=0x6c5800) at daemon/thread.c:304
#8  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc195ef3700 (LWP 2533)):
#0  0x00007fc19e2857a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc199a1ee5c in SyncObject::wait (this=0x9001d0, tv=...) at syncobject.hh:42
#2  0x00007fc199a1b397 in IdleTask::run (this=0x6f7880, d=...) at dispatcher.cc:244
#3  0x00007fc199a1c49c in Dispatcher::run (this=0x9001c0) at dispatcher.cc:119
#4  0x00007fc199a1c9fc in launch_dispatcher_thread (arg=0x9001c0) at dispatcher.cc:28
#5  0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc19dfe8ead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc1966f4700 (LWP 2532)):
#0  0x00007fc19df3a6b5 in raise () from /lib64/libc.so.6
#1  0x00007fc19df3be95 in abort () from /lib64/libc.so.6
#2  0x00007fc19df33645 in __assert_fail () from /lib64/libc.so.6
#3  0x00007fc199ac1096 in MCKVStore::commit (this=0xb6bb00) at mc-kvstore/mc-kvstore.cc:187
#4  0x00007fc199a2123e in TransactionContext::commit (this=0x7ecec8) at ep.cc:2631
#5  0x00007fc199a25ef8 in EventuallyPersistentStore::flushSome (this=0x7ecd00, q=0x7ece30, rejectQueue=0x16d8370) at ep.cc:1936
#6  0x00007fc199a74f94 in Flusher::doFlush (this=0x7b87e0) at flusher.cc:325
#7  0x00007fc199a75626 in Flusher::step (this=0x7b87e0, d=..., tid=...) at flusher.cc:239
#8  0x00007fc199a75abf in FlusherStepper::callback (this=0x6c3df0, d=..., t=...) at flusher.cc:23
#9  0x00007fc199a1d8ee in Task::run (this=0x6f7c80, d=..., t=...) at dispatcher.hh:139
#10 0x00007fc199a1c49c in Dispatcher::run (this=0x900540) at dispatcher.cc:119
#11 0x00007fc199a1c9fc in launch_dispatcher_thread (arg=0x900540) at dispatcher.cc:28
#12 0x00007fc19e2817e1 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fc19dfe8ead in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Nov  7 21:40 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
13c89aae52d83681cef2a67531dbbc65  /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Nov  7 21:40 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
3a4104b298d23d059d6b2fbc5746ef98  /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 13 Feb 24  2011 /lib64/libdl.so.2 -> libdl-2.12.so
643aa33f4e56d9dc502a3915703335a0  /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 12 Feb 24  2011 /lib64/libm.so.6 -> libm-2.12.so
b496148ca454ca96f9e39c88730b2f7a  /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 13 Feb 24  2011 /lib64/librt.so.1 -> librt-2.12.so
d399fe4ea992cffb7227bbbc5dd827e6  /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.0:
lrwxrwxrwx 1 bin bin 28 Nov  7 21:40 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0
4c88156c56aabbd2a6474b31a6fe21ef  /opt/couchbase/lib/libtcmalloc_minimal.so.0
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 18 Feb 24  2011 /lib64/libpthread.so.0 -> libpthread-2.12.so
e3cc21f2a81e11035cfadb97580867ef  /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 12 Feb 24  2011 /lib64/libc.so.6 -> libc-2.12.so
b3e39b32eddb15662ba7d1c700a65bf0  /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 10 Feb 24  2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so
bdfe53856f4614f5fb1a4fcad54d235c  /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 Feb 24  2011 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13
0eb10e4f59017e1796fc0b1e227b9467  /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Feb 24  2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1
704445fcf4f7f80bf04c446957c31862  /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 15410 Nov  6 01:10 /opt/couchbase/lib/memcached/stdin_term_handler.so
61e9fec1f8429c34a8a105df0c928631  /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Nov  7 21:40 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
9f536ac339f17a605f874c4739c79fae  /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Nov  7 21:40 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
3e914d5acdda8c4dc57aee546569ee34  /opt/couchbase/lib/memcached/ep.so
1.7.1.1_205_ge1a9877
CEP engine v1.7.1.1_205_ge1a9877
/lib64/libnss_files.so.2:
lrwxrwxrwx 1 root root 20 Feb 24  2011 /lib64/libnss_files.so.2 -> libnss_files-2.12.so
f4236af9b9a52a21f4d5e3fb7b865590  /lib64/libnss_files.so.2