Basic crash dump analysis of /tmp//core.memcached.15445. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 442917 May 5 17:47 /opt/couchbase/./bin/memcached c7eeca8ce7f76a2acaddffa809066a2b /opt/couchbase/./bin/memcached memcached 1.4.4_542_g19901a1 memcached 1.4.4_542_g19901a1 VERSION 1.4.4_542_g19901a1 -rw------- 1 couchbase couchbase 242204672 May 4 06:18 /tmp//core.memcached.15445 f72f19306c77ccb9ca10cdaf3dd81d83 /tmp//core.memcached.15445 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) CentOS (7.0.1-42.el5.centos) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. warning: exec file is newer than core file. [New Thread 15463] [New Thread 15465] [New Thread 15464] [New Thread 15462] [New Thread 15461] [New Thread 15460] [New Thread 15459] [New Thread 15458] [New Thread 15457] [New Thread 15456] [New Thread 15455] [New Thread 15454] [New Thread 15445] warning: .dynamic section for "/opt/couchbase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?) Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done. Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1 Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done. Loaded symbols for /opt/couchbase/lib/libsnappy.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff15fd000 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 0x00002aaaaad48f37 in std::_Rb_tree, std::allocator > > >, std::_Select1st, std::allocator > > > >, std::less, std::allocator, std::allocator > > > > >::_M_erase(std::_Rb_tree_node, std::allocator > > > >*) () from /opt/couchbase/lib/memcached/ep.so Thread 13 (Thread 0x2aac6867f240 (LWP 15445)): #0 0x00002aac67eeb8a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aac670f8ba6 in epoll_dispatch (base=0x1dcf6000, tv=) at epoll.c:404 #2 0x00002aac670e7c11 in event_base_loop (base=0x1dcf6000, flags=0) at event.c:1558 #3 0x00000000004095b2 in main (argc=, argv=) at daemon/memcached.c:7900 Thread 12 (Thread 15454): #0 0x00002aac67edd78b in read () from /lib64/libc.so.6 #1 0x00002aac67e83d57 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00002aac67e8471e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00002aac67e7ffdb in getc () from /lib64/libc.so.6 #4 0x00002aac68680879 in check_stdin_thread (arg=0x4033f0) at extensions/daemon/stdin_check.c:19 #5 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #6 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 11 (Thread 15455): #0 0x00002aac67eb1541 in nanosleep () from /lib64/libc.so.6 #1 0x00002aac67eb1364 in sleep () from /lib64/libc.so.6 #2 0x0000000000415c58 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #4 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 10 (Thread 15456): #0 0x00002aac67eeb8a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aac670f8ba6 in epoll_dispatch (base=0x1dcf6500, tv=) at epoll.c:404 #2 0x00002aac670e7c11 in event_base_loop (base=0x1dcf6500, flags=0) at event.c:1558 #3 0x0000000000414074 in worker_libevent (arg=0x19896500) at daemon/thread.c:305 #4 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #5 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 9 (Thread 15457): #0 0x00002aac67eeb8a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aac670f8ba6 in epoll_dispatch (base=0x1dcf6280, tv=) at epoll.c:404 #2 0x00002aac670e7c11 in event_base_loop (base=0x1dcf6280, flags=0) at event.c:1558 #3 0x0000000000414074 in worker_libevent (arg=0x19896600) at daemon/thread.c:305 #4 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #5 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 8 (Thread 15458): #0 0x00002aac67eeb8a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aac670f8ba6 in epoll_dispatch (base=0x1dcf6c80, tv=) at epoll.c:404 #2 0x00002aac670e7c11 in event_base_loop (base=0x1dcf6c80, flags=0) at event.c:1558 #3 0x0000000000414074 in worker_libevent (arg=0x19896700) at daemon/thread.c:305 #4 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #5 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 7 (Thread 15459): #0 0x00002aac67eeb8a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aac670f8ba6 in epoll_dispatch (base=0x1dcf6a00, tv=) at epoll.c:404 #2 0x00002aac670e7c11 in event_base_loop (base=0x1dcf6a00, flags=0) at event.c:1558 #3 0x0000000000414074 in worker_libevent (arg=0x19896800) at daemon/thread.c:305 #4 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #5 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 6 (Thread 15460): #0 0x00002aac67eeb8a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aac670f8ba6 in epoll_dispatch (base=0x1dcf6780, tv=) at epoll.c:404 #2 0x00002aac670e7c11 in event_base_loop (base=0x1dcf6780, flags=0) at event.c:1558 #3 0x0000000000414074 in worker_libevent (arg=0x19896900) at daemon/thread.c:305 #4 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #5 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 5 (Thread 15461): #0 0x00002aac67eb1541 in nanosleep () from /lib64/libc.so.6 #1 0x00002aac67ee4ed4 in usleep () from /lib64/libc.so.6 #2 0x00002aaaaad6c9a2 in MemoryTracker::getInstance () at memory_tracker.cc:37 #3 0x0000000000000000 in ?? () Thread 4 (Thread 15462): #0 0x00002aac67c06ee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad2d8c7 in wait (this=0x1dd40a80) at syncobject.hh:36 #2 Dispatcher::run (this=0x1dd40a80) at dispatcher.cc:89 #3 0x00002aaaaad2defb in launch_dispatcher_thread (arg=0x1dd40a80) at dispatcher.cc:28 #4 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #5 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 3 (Thread 15464): #0 0x00002aac67c07150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad2b080 in wait (this=0x1dd5e080, d=...) at syncobject.hh:47 #2 IdleTask::run (this=0x1dd5e080, d=...) at dispatcher.cc:294 #3 0x00002aaaaad2d670 in Dispatcher::run (this=0x1dd41880) at dispatcher.cc:123 #4 0x00002aaaaad2defb in launch_dispatcher_thread (arg=0x1dd41880) at dispatcher.cc:28 #5 0x00002aac67c0273d in start_thread () from /lib64/libpthread.so.0 #6 0x00002aac67eeb4bd in clone () from /lib64/libc.so.6 Thread 2 (Thread 15465): #0 0x00002aac67c07150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad4b39f in advance_tv (this=0x1dd4c000) at common.hh:81 #2 wait (this=0x1dd4c000) at syncobject.hh:62 #3 wait (this=0x1dd4c000) at tapconnmap.hh:171 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x1dd4c000) at ep_engine.cc:3382 #5 0x00002aaaaad4b483 in wait (this=) at syncobject.hh:55 #6 wait (this=) at syncobject.hh:63 #7 wait (this=) at tapconnmap.hh:171 #8 EventuallyPersistentEngine::notifyPendingConnections ( this=) at ep_engine.cc:3382 #9 0x0000000000000000 in ?? () Thread 1 (Thread 0x46bcd940 (LWP 15463)): #0 0x00002aaaaad48f37 in std::_Rb_tree, std::allocator > > >, std::_Select1st, std::allocator > > > >, std::less, std::allocator, std::allocator > > > > >::_M_erase(std::_Rb_tree_node, std::allocator > > > >*) () from /opt/couchbase/lib/memcached/ep.so #1 0x0000000020226a18 in ?? () #2 0x0000000000000001 in ?? () #3 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 May 7 08:03 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 f3ac72f589c7e3b6d2bbef2aef586372 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 /opt/couchbase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 May 7 08:03 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 63e107d5da485dcf41fa2bc5f551e2d6 /opt/couchbase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 12 Sep 14 2011 /lib64/libdl.so.2 -> libdl-2.5.so 47fe8c8b383dca4992313fb8edfe4a89 /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 11 Sep 14 2011 /lib64/libm.so.6 -> libm-2.5.so 70ce2bcde4aef3579e610c4b6d0e61a7 /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 12 Sep 14 2011 /lib64/librt.so.1 -> librt-2.5.so adf7d33b39f7051854e854723f672678 /lib64/librt.so.1 /opt/couchbase/lib/libtcmalloc_minimal.so.4: lrwxrwxrwx 1 bin bin 28 May 7 08:03 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0 66d6b566eeb60ed9a30d498192e1926f /opt/couchbase/lib/libtcmalloc_minimal.so.4 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 17 Sep 14 2011 /lib64/libpthread.so.0 -> libpthread-2.5.so 06b43304a38330ba4a928d2c9584744b /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 11 Sep 14 2011 /lib64/libc.so.6 -> libc-2.5.so 40a4baf51f54d1d9b38c3134bfce4fc0 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 9 Sep 14 2011 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so 8df7cc56ac89d47ec6e755ea5aba6ec3 /lib64/ld-linux-x86-64.so.2 /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 18 Sep 14 2011 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.8 11280e717c082632a690edc9349aea5c /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Sep 14 2011 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1 7dc360bdfd297bc69eb679e083037342 /lib64/libgcc_s.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 17082 May 5 17:47 /opt/couchbase/lib/memcached/stdin_term_handler.so f3fe897ecd49c47fc3ab0c14a04a8f6a /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 May 7 08:03 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 434d00f3ed4f927264211c16f7477d39 /opt/couchbase/lib/memcached/bucket_engine.so /opt/couchbase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 May 7 08:03 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0 b5002d544b9d13e5aef0195a9bccad39 /opt/couchbase/lib/memcached/ep.so /opt/couchbase/lib/libcouchstore.so.1: lrwxrwxrwx 1 bin bin 22 May 7 08:03 /opt/couchbase/lib/libcouchstore.so.1 -> libcouchstore.so.1.0.0 632f32f8ba9d808f047a40c172cfb459 /opt/couchbase/lib/libcouchstore.so.1 /opt/couchbase/lib/libsnappy.so.1: lrwxrwxrwx 1 bin bin 18 May 7 08:03 /opt/couchbase/lib/libsnappy.so.1 -> libsnappy.so.1.1.2 7112708b58811a83581368abce38084c /opt/couchbase/lib/libsnappy.so.1 /lib64/libnss_files.so.2: lrwxrwxrwx 1 root root 19 Sep 14 2011 /lib64/libnss_files.so.2 -> libnss_files-2.5.so 05edfb87fbaaeff2fe6eb90b0a670e2b /lib64/libnss_files.so.2