Basic crash dump analysis of /tmp/core.memcached.2422.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 396049 Nov  6 01:09 /opt/couchbase/bin/memcached
a7af118943eb2f6bf525ac17a3ff1a7d  /opt/couchbase/bin/memcached
-rw------- 1 couchbase couchbase 254332928 Nov  7 22:24 /tmp/core.memcached.2422
89b4c5e62aeae920754e34801480f092  /tmp/core.memcached.2422
ep_version1.7.1.1_205_ge1a9877
ep_version1.7.1.1_205_ge1a9877
1.7.1.1_205_ge1a9877
ep_version1.7.1.1_205_ge1a9877
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) Amazon Linux (7.2-48.10.amzn1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 2571]
[New Thread 2429]
[New Thread 2572]
[New Thread 2573]
[New Thread 2422]
[New Thread 2428]
[New Thread 2434]
[New Thread 2508]
[New Thread 2523]
[New Thread 2552]
[New Thread 2561]
[New Thread 2570]
[New Thread 2430]
[New Thread 2432]
[New Thread 2433]
[New Thread 2431]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0  0x00007f7789a986b5 in raise () from /lib64/libc.so.6

Thread 16 (Thread 0x7f778793b700 (LWP 2431)):
#0  0x00007f7789b474a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f778a8f1c28 in epoll_dispatch (base=0x1c6e280, tv=<value optimized out>) at epoll.c:404
#2  0x00007f778a8e0a4c in event_base_loop (base=0x1c6e280, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x1c50600) at daemon/thread.c:304
#4  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f7786939700 (LWP 2433)):
#0  0x00007f7789b474a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f778a8f1c28 in epoll_dispatch (base=0x1c6ea00, tv=<value optimized out>) at epoll.c:404
#2  0x00007f778a8e0a4c in event_base_loop (base=0x1c6ea00, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x1c50800) at daemon/thread.c:304
#4  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f778713a700 (LWP 2432)):
#0  0x00007f7789b474a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f778a8f1c28 in epoll_dispatch (base=0x1c6ec80, tv=<value optimized out>) at epoll.c:404
#2  0x00007f778a8e0a4c in event_base_loop (base=0x1c6ec80, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x1c50700) at daemon/thread.c:304
#4  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f778813c700 (LWP 2430)):
#0  0x00007f7789b474a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f778a8f1c28 in epoll_dispatch (base=0x1c6e500, tv=<value optimized out>) at epoll.c:404
#2  0x00007f778a8e0a4c in event_base_loop (base=0x1c6e500, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x1c50500) at daemon/thread.c:304
#4  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f7781250700 (LWP 2570)):
#0  0x00007f7789de343c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f778557de16 in SyncObject::wait (this=0x1fa98d0) at syncobject.hh:31
#2  0x00007f778557a1bd in Dispatcher::run (this=0x1fa98c0) at dispatcher.cc:85
#3  0x00007f778557a9fc in launch_dispatcher_thread (arg=0x1fa98c0) at dispatcher.cc:28
#4  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f7785258700 (LWP 2561)):
#0  0x00007f7789de37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f77881407e7 in engine_shutdown_thread (arg=0x1c73240) at bucket_engine.c:1432
#2  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f778024e700 (LWP 2552)):
#0  0x00007f7789de37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f77881407e7 in engine_shutdown_thread (arg=0x1c73600) at bucket_engine.c:1432
#2  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f7782a53700 (LWP 2523)):
#0  0x00007f7789de37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f77881407e7 in engine_shutdown_thread (arg=0x1c736c0) at bucket_engine.c:1432
#2  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f7783254700 (LWP 2508)):
#0  0x00007f7789de37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f77881407e7 in engine_shutdown_thread (arg=0x1c730c0) at bucket_engine.c:1432
#2  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f7786138700 (LWP 2434)):
#0  0x00007f7789b474a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f778a8f1c28 in epoll_dispatch (base=0x1c6e780, tv=<value optimized out>) at epoll.c:404
#2  0x00007f778a8e0a4c in event_base_loop (base=0x1c6e780, flags=0) at event.c:1558
#3  0x00000000004134a4 in worker_libevent (arg=0x1c50900) at daemon/thread.c:304
#4  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f7789347700 (LWP 2428)):
#0  0x00007f7789b39b5d in read () from /lib64/libc.so.6
#1  0x00007f7789ad7228 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x00007f7789ad8d2e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3  0x00007f7789ad424b in getc () from /lib64/libc.so.6
#4  0x00007f7789348875 in check_stdin_thread (arg=0x403610) at extensions/daemon/stdin_check.c:19
#5  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f778af26720 (LWP 2422)):
#0  0x00007f7789b474a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f778a8f1c28 in epoll_dispatch (base=0x1c6e000, tv=<value optimized out>) at epoll.c:404
#2  0x00007f778a8e0a4c in event_base_loop (base=0x1c6e000, flags=0) at event.c:1558
#3  0x000000000040930b in main (argc=15, argv=0x7ffff210da98) at daemon/memcached.c:7531

Thread 4 (Thread 0x7f7780a4f700 (LWP 2573)):
#0  0x00007f7789de37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f778557ce5c in SyncObject::wait (this=0x1c70500, tv=...) at syncobject.hh:42
#2  0x00007f77855c6a9c in SyncObject::wait (this=0x1c70500, secs=1) at syncobject.hh:58
#3  0x00007f77855c6ad7 in TapConnMap::wait (this=0x1c70500, howlong=1) at tapconnmap.hh:209
#4  0x00007f77855aa04f in EventuallyPersistentEngine::notifyTapIoThread (this=0x222e000) at ep_engine.cc:3393
#5  0x00007f77855b3224 in EvpNotifyTapIo (arg=0x222e000) at ep_engine.cc:1037
#6  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f7781a51700 (LWP 2572)):
#0  0x00007f7789de37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f778557ce5c in SyncObject::wait (this=0x1fa9710, tv=...) at syncobject.hh:42
#2  0x00007f7785579397 in IdleTask::run (this=0x1e88d80, d=...) at dispatcher.cc:244
#3  0x00007f778557a49c in Dispatcher::run (this=0x1fa9700) at dispatcher.cc:119
#4  0x00007f778557a9fc in launch_dispatcher_thread (arg=0x1fa9700) at dispatcher.cc:28
#5  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f7788b46700 (LWP 2429)):
#0  0x00007f7789b0c46d in nanosleep () from /lib64/libc.so.6
#1  0x00007f7789b0c2e0 in sleep () from /lib64/libc.so.6
#2  0x0000000000415068 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
#3  0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f7789b46ead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f7782252700 (LWP 2571)):
#0  0x00007f7789a986b5 in raise () from /lib64/libc.so.6
#1  0x00007f7789a99e95 in abort () from /lib64/libc.so.6
#2  0x00007f7789a91645 in __assert_fail () from /lib64/libc.so.6
#3  0x00007f778561f096 in MCKVStore::commit (this=0x214a740) at mc-kvstore/mc-kvstore.cc:187
#4  0x00007f778557f23e in TransactionContext::commit (this=0x1c66208) at ep.cc:2631
#5  0x00007f7785583ef8 in EventuallyPersistentStore::flushSome (this=0x1c66040, q=0x1c66170, rejectQueue=0x33f6eb0) at ep.cc:1936
#6  0x00007f77855d2f94 in Flusher::doFlush (this=0x1d466c0) at flusher.cc:325
#7  0x00007f77855d3626 in Flusher::step (this=0x1d466c0, d=..., tid=...) at flusher.cc:239
#8  0x00007f77855d3abf in FlusherStepper::callback (this=0x1c4ef10, d=..., t=...) at flusher.cc:23
#9  0x00007f778557b8ee in Task::run (this=0x1e88600, d=..., t=...) at dispatcher.hh:139
#10 0x00007f778557a49c in Dispatcher::run (this=0x1fa9a80) at dispatcher.cc:119
#11 0x00007f778557a9fc in launch_dispatcher_thread (arg=0x1fa9a80) at dispatcher.cc:28
#12 0x00007f7789ddf7e1 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f7789b46ead in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Nov  7 21:40 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
13c89aae52d83681cef2a67531dbbc65  /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Nov  7 21:40 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
3a4104b298d23d059d6b2fbc5746ef98  /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 13 Feb 24  2011 /lib64/libdl.so.2 -> libdl-2.12.so
643aa33f4e56d9dc502a3915703335a0  /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 12 Feb 24  2011 /lib64/libm.so.6 -> libm-2.12.so
b496148ca454ca96f9e39c88730b2f7a  /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 13 Feb 24  2011 /lib64/librt.so.1 -> librt-2.12.so
d399fe4ea992cffb7227bbbc5dd827e6  /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.0:
lrwxrwxrwx 1 bin bin 28 Nov  7 21:40 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0
4c88156c56aabbd2a6474b31a6fe21ef  /opt/couchbase/lib/libtcmalloc_minimal.so.0
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 18 Feb 24  2011 /lib64/libpthread.so.0 -> libpthread-2.12.so
e3cc21f2a81e11035cfadb97580867ef  /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 12 Feb 24  2011 /lib64/libc.so.6 -> libc-2.12.so
b3e39b32eddb15662ba7d1c700a65bf0  /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 10 Feb 24  2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so
bdfe53856f4614f5fb1a4fcad54d235c  /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 Feb 24  2011 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13
0eb10e4f59017e1796fc0b1e227b9467  /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Feb 24  2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1
704445fcf4f7f80bf04c446957c31862  /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 15410 Nov  6 01:10 /opt/couchbase/lib/memcached/stdin_term_handler.so
61e9fec1f8429c34a8a105df0c928631  /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Nov  7 21:40 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
9f536ac339f17a605f874c4739c79fae  /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Nov  7 21:40 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
3e914d5acdda8c4dc57aee546569ee34  /opt/couchbase/lib/memcached/ep.so
1.7.1.1_205_ge1a9877
CEP engine v1.7.1.1_205_ge1a9877
/lib64/libnss_files.so.2:
lrwxrwxrwx 1 root root 20 Feb 24  2011 /lib64/libnss_files.so.2 -> libnss_files-2.12.so
f4236af9b9a52a21f4d5e3fb7b865590  /lib64/libnss_files.so.2