Basic crash dump analysis of /tmp//core.memcached.4032.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 442917 Apr 30 17:07 /opt/couchbase/./bin/memcached
c7eeca8ce7f76a2acaddffa809066a2b  /opt/couchbase/./bin/memcached
memcached 1.4.4_542_g19901a1
memcached 1.4.4_542_g19901a1
VERSION 1.4.4_542_g19901a1
-rw------- 1 couchbase couchbase 279183360 Apr 30 21:05 /tmp//core.memcached.4032
b339d160f4efe124801226baac08de98  /tmp//core.memcached.4032
version1.4.4_542_g19901a1
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 4573]
[New Thread 4051]
[New Thread 4050]
[New Thread 4049]
[New Thread 4048]
[New Thread 4047]
[New Thread 4046]
[New Thread 4045]
[New Thread 4044]
[New Thread 4043]
[New Thread 4042]
[New Thread 4041]
[New Thread 4032]
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffe8ef0000
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0  0x00002b759b2ab265 in raise () from /lib64/libc.so.6

Thread 13 (Thread 0x2b759bae3240 (LWP 4032)):
#0  0x00002b759b34f8a8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b759a55cba6 in epoll_dispatch (base=0x21562000, 
    tv=<value optimized out>) at epoll.c:404
#2  0x00002b759a54bc11 in event_base_loop (base=0x21562000, flags=0)
    at event.c:1558
#3  0x00000000004095b2 in main (argc=<value optimized out>, 
    argv=<value optimized out>) at daemon/memcached.c:7900

Thread 12 (Thread 4041):
#0  0x00002b759b34178b in read () from /lib64/libc.so.6
#1  0x00002b759b2e7d57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x00002b759b2e871e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3  0x00002b759b2e3fdb in getc () from /lib64/libc.so.6
#4  0x00002b759bae4879 in check_stdin_thread (arg=0x4033f0)
    at extensions/daemon/stdin_check.c:19
#5  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#6  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 11 (Thread 4042):
#0  0x00002b759b315541 in nanosleep () from /lib64/libc.so.6
#1  0x00002b759b315364 in sleep () from /lib64/libc.so.6
#2  0x0000000000415c58 in check_isasl_db_thread (arg=<value optimized out>)
    at daemon/isasl.c:233
#3  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#4  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 4043):
#0  0x00002b759b34f8a8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b759a55cba6 in epoll_dispatch (base=0x21562500, 
    tv=<value optimized out>) at epoll.c:404
#2  0x00002b759a54bc11 in event_base_loop (base=0x21562500, flags=0)
    at event.c:1558
#3  0x0000000000414074 in worker_libevent (arg=0x1d102500)
    at daemon/thread.c:305
#4  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#5  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 9 (Thread 4044):
#0  0x00002b759b34f8a8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b759a55cba6 in epoll_dispatch (base=0x21562280, 
    tv=<value optimized out>) at epoll.c:404
#2  0x00002b759a54bc11 in event_base_loop (base=0x21562280, flags=0)
    at event.c:1558
#3  0x0000000000414074 in worker_libevent (arg=0x1d102600)
    at daemon/thread.c:305
#4  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#5  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 4045):
#0  0x00002b759b34f8a8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b759a55cba6 in epoll_dispatch (base=0x21562c80, 
    tv=<value optimized out>) at epoll.c:404
#2  0x00002b759a54bc11 in event_base_loop (base=0x21562c80, flags=0)
    at event.c:1558
#3  0x0000000000414074 in worker_libevent (arg=0x1d102700)
    at daemon/thread.c:305
#4  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#5  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 4046):
#0  0x00002b759b34f8a8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b759a55cba6 in epoll_dispatch (base=0x21562a00, 
    tv=<value optimized out>) at epoll.c:404
#2  0x00002b759a54bc11 in event_base_loop (base=0x21562a00, flags=0)
    at event.c:1558
#3  0x0000000000414074 in worker_libevent (arg=0x1d102800)
    at daemon/thread.c:305
#4  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#5  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 4047):
#0  0x00002b759b34f8a8 in epoll_wait () from /lib64/libc.so.6
#1  0x00002b759a55cba6 in epoll_dispatch (base=0x21562780, 
    tv=<value optimized out>) at epoll.c:404
#2  0x00002b759a54bc11 in event_base_loop (base=0x21562780, flags=0)
    at event.c:1558
#3  0x0000000000414074 in worker_libevent (arg=0x1d102900)
    at daemon/thread.c:305
#4  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#5  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 4048):
#0  0x00002b759b315541 in nanosleep () from /lib64/libc.so.6
#1  0x00002b759b348ed4 in usleep () from /lib64/libc.so.6
#2  0x00002aaaaad6c692 in updateStatsThread (arg=0x1d1007c0)
    at memory_tracker.cc:31
#3  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#4  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 4049):
#0  0x00002b759b06aee9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002aaaaad2d7f7 in wait (this=0x215c8a80) at syncobject.hh:36
#2  Dispatcher::run (this=0x215c8a80) at dispatcher.cc:89
#3  0x00002aaaaad2de2b in launch_dispatcher_thread (arg=0x215c8a80)
    at dispatcher.cc:28
#4  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#5  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 4050):
#0  0x00002b759b06b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002aaaaad2afb0 in wait (this=0x215e6000, d=...) at syncobject.hh:47
#2  IdleTask::run (this=0x215e6000, d=...) at dispatcher.cc:294
#3  0x00002aaaaad2d5a0 in Dispatcher::run (this=0x215c8c40)
    at dispatcher.cc:123
#4  0x00002aaaaad2de2b in launch_dispatcher_thread (arg=0x215c8c40)
    at dispatcher.cc:28
#5  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#6  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 4051):
#0  0x00002b759b06b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002aaaaad2afb0 in wait (this=0x215e6080, d=...) at syncobject.hh:47
#2  IdleTask::run (this=0x215e6080, d=...) at dispatcher.cc:294
#3  0x00002aaaaad2d5a0 in Dispatcher::run (this=0x215c9880)
    at dispatcher.cc:123
#4  0x00002aaaaad2de2b in launch_dispatcher_thread (arg=0x215c9880)
    at dispatcher.cc:28
#5  0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#6  0x00002b759b34f4bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x486b8940 (LWP 4573)):
#0  0x00002b759b2ab265 in raise () from /lib64/libc.so.6
#1  0x00002b759b2acd10 in abort () from /lib64/libc.so.6
#2  0x00002b759b2a46e6 in __assert_fail () from /lib64/libc.so.6
#3  0x00002aaaaaab1a30 in do_bucket_engine_release_cookie (cookie=0x21508840)
    at bucket_engine.c:2488
#4  bucket_engine_release_cookie (cookie=0x21508840) at bucket_engine.c:2530
#5  0x00002aaaaad7c93f in TapConnection::releaseReference (this=0x215e8400, 
    force=<value optimized out>) at tapconnection.cc:52
#6  0x00002aaaaad8f6e8 in TapConnectionReaperCallback::TapConnectionReaperCallback(EventuallyPersistentEngine&, TapConnection*) ()
   from /opt/couchbase/lib/memcached/ep.so
#7  0x00002aaaaad8d948 in TapConnMap::shutdownAllTapConnections (
    this=0x215ec000) at tapconnmap.cc:392
#8  0x00002aaaaad4cf4c in EventuallyPersistentEngine::destroy (
    this=0x215d4000, force=<value optimized out>) at ep_engine.cc:1332
#9  0x00002aaaaad4f58e in EvpDestroy (handle=<value optimized out>, 
    force=true) at ep_engine.cc:123
#10 0x00002aaaaaaaf3f6 in engine_shutdown_thread (arg=0x215682a0)
    at bucket_engine.c:1364
#11 0x00002b759b06673d in start_thread () from /lib64/libpthread.so.0
#12 0x00002b759b34f4bd in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Apr 30 18:02 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
f3ac72f589c7e3b6d2bbef2aef586372  /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Apr 30 18:02 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
63e107d5da485dcf41fa2bc5f551e2d6  /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 12 Sep 14  2011 /lib64/libdl.so.2 -> libdl-2.5.so
47fe8c8b383dca4992313fb8edfe4a89  /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 11 Sep 14  2011 /lib64/libm.so.6 -> libm-2.5.so
70ce2bcde4aef3579e610c4b6d0e61a7  /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 12 Sep 14  2011 /lib64/librt.so.1 -> librt-2.5.so
adf7d33b39f7051854e854723f672678  /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.4:
lrwxrwxrwx 1 bin bin 28 Apr 30 18:02 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0
66d6b566eeb60ed9a30d498192e1926f  /opt/couchbase/lib/libtcmalloc_minimal.so.4
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 17 Sep 14  2011 /lib64/libpthread.so.0 -> libpthread-2.5.so
06b43304a38330ba4a928d2c9584744b  /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 11 Sep 14  2011 /lib64/libc.so.6 -> libc-2.5.so
40a4baf51f54d1d9b38c3134bfce4fc0  /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 9 Sep 14  2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so
8df7cc56ac89d47ec6e755ea5aba6ec3  /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 18 Mar 15 04:20 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.8
f21f4163970f603a6d6e5ae781bfcd3b  /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Mar 15 04:20 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1
79a8da27c719564899e177ec7189de5e  /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 17082 Apr 30 17:07 /opt/couchbase/lib/memcached/stdin_term_handler.so
f3fe897ecd49c47fc3ab0c14a04a8f6a  /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Apr 30 18:02 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
434d00f3ed4f927264211c16f7477d39  /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Apr 30 18:02 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
6b15329ff915753befe17b054bf533e8  /opt/couchbase/lib/memcached/ep.so
/opt/couchbase/lib/libcouchstore.so.1:
lrwxrwxrwx 1 bin bin 22 Apr 30 18:02 /opt/couchbase/lib/libcouchstore.so.1 -> libcouchstore.so.1.0.0
6abc1c5821f4576f92903d0df4bf4540  /opt/couchbase/lib/libcouchstore.so.1
/opt/couchbase/lib/libsnappy.so.1:
lrwxrwxrwx 1 bin bin 18 Apr 30 18:02 /opt/couchbase/lib/libsnappy.so.1 -> libsnappy.so.1.1.2
7112708b58811a83581368abce38084c  /opt/couchbase/lib/libsnappy.so.1
/lib64/libnss_files.so.2:
lrwxrwxrwx 1 root root 19 Sep 14  2011 /lib64/libnss_files.so.2 -> libnss_files-2.5.so
05edfb87fbaaeff2fe6eb90b0a670e2b  /lib64/libnss_files.so.2