Basic crash dump analysis of /tmp/core.memcached.2406. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 396049 Nov 6 01:09 /opt/couchbase/bin/memcached a7af118943eb2f6bf525ac17a3ff1a7d /opt/couchbase/bin/memcached -rw------- 1 couchbase couchbase 135712768 Nov 7 22:21 /tmp/core.memcached.2406 e3af1994fedc83e6f00a5326c4d8995e /tmp/core.memcached.2406 1.7.1.1_205_ge1a9877 ep_version1.7.1.1_205_ge1a9877 ep_version1.7.1.1_205_ge1a9877 ep_version1.7.1.1_205_ge1a9877 ep_version1.7.1.1_205_ge1a9877 ep_version1.7.1.1_205_ge1a9877 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Amazon Linux (7.2-48.10.amzn1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-amazon-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 2529] [New Thread 2531] [New Thread 2530] [New Thread 2465] [New Thread 2406] [New Thread 2519] [New Thread 2415] [New Thread 2480] [New Thread 2417] [New Thread 2509] [New Thread 2421] [New Thread 2416] [New Thread 2528] [New Thread 2418] [New Thread 2420] [New Thread 2419] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 6, Aborted. #0 0x00007f61409836b5 in raise () from /lib64/libc.so.6 Thread 16 (Thread 0x7f613e025700 (LWP 2419)): #0 0x00007f6140a324a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f61417dcc28 in epoll_dispatch (base=0x218dc80, tv=) at epoll.c:404 #2 0x00007f61417cba4c in event_base_loop (base=0x218dc80, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x216f700) at daemon/thread.c:304 #4 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f613d824700 (LWP 2420)): #0 0x00007f6140a324a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f61417dcc28 in epoll_dispatch (base=0x218da00, tv=) at epoll.c:404 #2 0x00007f61417cba4c in event_base_loop (base=0x218da00, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x216f800) at daemon/thread.c:304 #4 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f613e826700 (LWP 2418)): #0 0x00007f613c442218 in AtomicPtr::operator bool () from /opt/couchbase/lib/memcached/ep.so #1 0x00007f613c45ea91 in RCPtr::operator bool (this=0x7f613e824ad0) at atomic.hh:370 #2 0x00007f613c45c2b3 in CheckpointManager::queueDirty (this=0x24c2680, qi=..., vbucket=...) at checkpoint.cc:531 #3 0x00007f613c46b67e in EventuallyPersistentStore::queueDirty (this=0x227dd00, key=..., vbid=214, op=queue_op_set, value=..., flags=0, exptime=0, cas=8854595084152, seqno=1, rowid=-1, tapBackfill=false) at ep.cc:2401 #4 0x00007f613c46cd0a in EventuallyPersistentStore::set (this=0x227dd00, itm=..., cookie=0x2640b88, force=true) at ep.cc:868 #5 0x00007f613c49c7d7 in EventuallyPersistentEngine::tapNotify(const void *, void *, uint16_t, uint8_t, uint16_t, ._169, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, const void *, size_t, uint16_t) (this=0x26ec000, cookie=0x2640b88, engine_specific=0x23c5020, nengine=22, tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=132, key=0x23c503e, nkey=11, flags=0, exptime=0, data=0x23c5049, ndata=655, vbucket=214) at ep_engine.cc:2031 #6 0x00007f613c49de32 in EvpTapNotify(ENGINE_HANDLE *, const void *, void *, uint16_t, uint8_t, uint16_t, ._169, uint32_t, const void *, size_t, uint32_t, uint32_t, uint64_t, const void *, size_t, uint16_t) (handle=0x26ec000, cookie=0x2640b88, engine_specific=0x23c5020, nengine=22, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=132, key=0x23c503e, nkey=11, flags=0, exptime=0, cas=8849460209339, data=0x23c5049, ndata=655, vbucket=214) at ep_engine.cc:960 #7 0x00007f613f02c56b in bucket_tap_notify (handle=0x7f613f230ba0, cookie=0x2640b88, engine_specific=0x23c5020, nengine=22, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=132, key=0x23c503e, nkey=11, flags=0, exptime=0, cas=8849460209339, data=0x23c5049, ndata=655, vbucket=214) at bucket_engine.c:1912 #8 0x0000000000406a11 in process_bin_tap_packet (event=36348208, c=0x2640b88) at daemon/memcached.c:2744 #9 0x0000000000410c05 in complete_nread_binary (c=0x7f613e824ad0) at daemon/memcached.c:3427 #10 0x00000000004118a1 in complete_nread (c=0x2640b88) at daemon/memcached.c:3509 #11 conn_nread (c=0x2640b88) at daemon/memcached.c:5319 #12 0x0000000000407564 in event_handler (fd=, which=, arg=0x2640b88) at daemon/memcached.c:5620 #13 0x00007f61417cbdf9 in event_process_active_single_queue (base=0x218d280, flags=0) at event.c:1308 #14 event_process_active (base=0x218d280, flags=0) at event.c:1375 #15 event_base_loop (base=0x218d280, flags=0) at event.c:1572 #16 0x00000000004134a4 in worker_libevent (arg=0x216f600) at daemon/thread.c:304 #17 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #18 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f613813b700 (LWP 2528)): #0 0x00007f6140cce43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613c468e16 in SyncObject::wait (this=0x24c7390) at syncobject.hh:31 #2 0x00007f613c4651bd in Dispatcher::run (this=0x24c7380) at dispatcher.cc:85 #3 0x00007f613c4659fc in launch_dispatcher_thread (arg=0x24c7380) at dispatcher.cc:28 #4 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f613fa31700 (LWP 2416)): #0 0x00007f61409f746d in nanosleep () from /lib64/libc.so.6 #1 0x00007f61409f72e0 in sleep () from /lib64/libc.so.6 #2 0x0000000000415068 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f613d023700 (LWP 2421)): #0 0x00007f6140a324a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f61417dcc28 in epoll_dispatch (base=0x218d780, tv=) at epoll.c:404 #2 0x00007f61417cba4c in event_base_loop (base=0x218d780, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x216f900) at daemon/thread.c:304 #4 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f6137139700 (LWP 2509)): #0 0x00007f6140cce7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613f02b7e7 in engine_shutdown_thread (arg=0x2192840) at bucket_engine.c:1432 #2 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f613f027700 (LWP 2417)): #0 0x00007f6140a324a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f61417dcc28 in epoll_dispatch (base=0x218d500, tv=) at epoll.c:404 #2 0x00007f61417cba4c in event_base_loop (base=0x218d500, flags=0) at event.c:1558 #3 0x00000000004134a4 in worker_libevent (arg=0x216f500) at daemon/thread.c:304 #4 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f613993e700 (LWP 2480)): #0 0x00007f6140cce7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613f02b7e7 in engine_shutdown_thread (arg=0x21923c0) at bucket_engine.c:1432 #2 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f6140232700 (LWP 2415)): #0 0x00007f6140a24b5d in read () from /lib64/libc.so.6 #1 0x00007f61409c2228 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00007f61409c3d2e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00007f61409bf24b in getc () from /lib64/libc.so.6 #4 0x00007f6140233875 in check_stdin_thread (arg=0x403610) at extensions/daemon/stdin_check.c:19 #5 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f613c143700 (LWP 2519)): #0 0x00007f6140cce7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613f02b7e7 in engine_shutdown_thread (arg=0x2192240) at bucket_engine.c:1432 #2 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f6141e11720 (LWP 2406)): #0 0x00007f6140a324a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f61417dcc28 in epoll_dispatch (base=0x218d000, tv=) at epoll.c:404 #2 0x00007f61417cba4c in event_base_loop (base=0x218d000, flags=0) at event.c:1558 #3 0x000000000040930b in main (argc=15, argv=0x7fff9d32d8b8) at daemon/memcached.c:7531 Thread 4 (Thread 0x7f613a13f700 (LWP 2465)): #0 0x00007f6140cce7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613f02b7e7 in engine_shutdown_thread (arg=0x21920c0) at bucket_engine.c:1432 #2 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f613893c700 (LWP 2530)): #0 0x00007f6140cce7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613c467e5c in SyncObject::wait (this=0x24c71d0, tv=...) at syncobject.hh:42 #2 0x00007f613c464397 in IdleTask::run (this=0x24c8900, d=...) at dispatcher.cc:244 #3 0x00007f613c46549c in Dispatcher::run (this=0x24c71c0) at dispatcher.cc:119 #4 0x00007f613c4659fc in launch_dispatcher_thread (arg=0x24c71c0) at dispatcher.cc:28 #5 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f613793a700 (LWP 2531)): #0 0x00007f6140cce7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f613c467e5c in SyncObject::wait (this=0x218f800, tv=...) at syncobject.hh:42 #2 0x00007f613c4b1a9c in SyncObject::wait (this=0x218f800, secs=1) at syncobject.hh:58 #3 0x00007f613c4b1ad7 in TapConnMap::wait (this=0x218f800, howlong=1) at tapconnmap.hh:209 #4 0x00007f613c49504f in EventuallyPersistentEngine::notifyTapIoThread (this=0x26ec000) at ep_engine.cc:3393 #5 0x00007f613c49e224 in EvpNotifyTapIo (arg=0x26ec000) at ep_engine.cc:1037 #6 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f6140a31ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f613913d700 (LWP 2529)): #0 0x00007f61409836b5 in raise () from /lib64/libc.so.6 #1 0x00007f6140984e95 in abort () from /lib64/libc.so.6 #2 0x00007f614097c645 in __assert_fail () from /lib64/libc.so.6 #3 0x00007f613c50a096 in MCKVStore::commit (this=0x2629180) at mc-kvstore/mc-kvstore.cc:187 #4 0x00007f613c46a23e in TransactionContext::commit (this=0x227dec8) at ep.cc:2631 #5 0x00007f613c46eef8 in EventuallyPersistentStore::flushSome (this=0x227dd00, q=0x227de30, rejectQueue=0x258cd20) at ep.cc:1936 #6 0x00007f613c4bdf94 in Flusher::doFlush (this=0x2262480) at flusher.cc:325 #7 0x00007f613c4be626 in Flusher::step (this=0x2262480, d=..., tid=...) at flusher.cc:239 #8 0x00007f613c4beabf in FlusherStepper::callback (this=0x216dd80, d=..., t=...) at flusher.cc:23 #9 0x00007f613c4668ee in Task::run (this=0x24c8280, d=..., t=...) at dispatcher.hh:139 #10 0x00007f613c46549c in Dispatcher::run (this=0x24c7540) at dispatcher.cc:119 #11 0x00007f613c4659fc in launch_dispatcher_thread (arg=0x24c7540) at dispatcher.cc:28 #12 0x00007f6140cca7e1 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f6140a31ead in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 Nov 7 21:40 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 13c89aae52d83681cef2a67531dbbc65 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 /opt/couchbase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 Nov 7 21:40 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 3a4104b298d23d059d6b2fbc5746ef98 /opt/couchbase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 13 Feb 24 2011 /lib64/libdl.so.2 -> libdl-2.12.so 643aa33f4e56d9dc502a3915703335a0 /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 12 Feb 24 2011 /lib64/libm.so.6 -> libm-2.12.so b496148ca454ca96f9e39c88730b2f7a /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 13 Feb 24 2011 /lib64/librt.so.1 -> librt-2.12.so d399fe4ea992cffb7227bbbc5dd827e6 /lib64/librt.so.1 /opt/couchbase/lib/libtcmalloc_minimal.so.0: lrwxrwxrwx 1 bin bin 28 Nov 7 21:40 /opt/couchbase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.1.0 4c88156c56aabbd2a6474b31a6fe21ef /opt/couchbase/lib/libtcmalloc_minimal.so.0 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 18 Feb 24 2011 /lib64/libpthread.so.0 -> libpthread-2.12.so e3cc21f2a81e11035cfadb97580867ef /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 12 Feb 24 2011 /lib64/libc.so.6 -> libc-2.12.so b3e39b32eddb15662ba7d1c700a65bf0 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 10 Feb 24 2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so bdfe53856f4614f5fb1a4fcad54d235c /lib64/ld-linux-x86-64.so.2 /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 19 Feb 24 2011 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 0eb10e4f59017e1796fc0b1e227b9467 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Feb 24 2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1 704445fcf4f7f80bf04c446957c31862 /lib64/libgcc_s.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15410 Nov 6 01:10 /opt/couchbase/lib/memcached/stdin_term_handler.so 61e9fec1f8429c34a8a105df0c928631 /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 Nov 7 21:40 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 9f536ac339f17a605f874c4739c79fae /opt/couchbase/lib/memcached/bucket_engine.so /opt/couchbase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 Nov 7 21:40 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0 3e914d5acdda8c4dc57aee546569ee34 /opt/couchbase/lib/memcached/ep.so 1.7.1.1_205_ge1a9877 CEP engine v1.7.1.1_205_ge1a9877 /lib64/libnss_files.so.2: lrwxrwxrwx 1 root root 20 Feb 24 2011 /lib64/libnss_files.so.2 -> libnss_files-2.12.so f4236af9b9a52a21f4d5e3fb7b865590 /lib64/libnss_files.so.2