Basic crash dump analysis of /tmp//core.memcached.31930. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 392112 May 5 02:48 /opt/membase/./bin/memcached -rw------- 1 membase membase 1655160832 May 5 07:55 /tmp//core.memcached.31930 e394c496b0cd3ab8935bf0ce6da08e32 /opt/membase/./bin/memcached aa5e96343ea5fac9ff0c16ef0e01b9f7 /tmp//core.memcached.31930 memcached 1.4.4_458_g6b3e533 memcached 1.4.4_458_g6b3e533 VERSION 1.4.4_458_g6b3e533 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 1304] [New Thread 327] [New Thread 31939] [New Thread 31938] [New Thread 31937] [New Thread 31936] [New Thread 31933] [New Thread 31932] [New Thread 31931] [New Thread 31930] Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x0000003874230265 in raise () from /lib64/libc.so.6 Thread 11 (Thread 31930): #0 0x00000038742d44b8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aafd6c05c28 in epoll_dispatch (base=0xbbeca80, tv=) at epoll.c:404 #2 0x00002aafd6bf4a4c in event_base_loop (base=0xbbeca80, flags=0) at event.c:1558 #3 0x0000000000409974 in main (argc=, argv=) at daemon/memcached.c:7459 Thread 10 (Thread 31931): #0 0x00000038742c63db in read () from /lib64/libc.so.6 #1 0x000000387426cd57 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x000000387426d71e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x0000003874268fdb in getc () from /lib64/libc.so.6 #4 0x00002aafd6e39855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19 #5 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 9 (Thread 31932): #0 0x00000038742dfe8e in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00000038742770bd in _L_lock_5478 () from /lib64/libc.so.6 #2 0x0000003874271de3 in _int_free () from /lib64/libc.so.6 #3 0x00000038742728bb in free () from /lib64/libc.so.6 #4 0x0000000000414a92 in free_user_ht () at daemon/isasl.c:95 #5 load_user_db () at daemon/isasl.c:115 #6 0x0000000000414f02 in check_isasl_db_thread (arg=) at daemon/isasl.c:223 #7 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #8 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 8 (Thread 31933): #0 0x00000038742d44b8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aafd6c05c28 in epoll_dispatch (base=0xbbf8070, tv=) at epoll.c:404 #2 0x00002aafd6bf4a4c in event_base_loop (base=0xbbf8070, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xbbf7b10) at daemon/thread.c:304 #4 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 7 (Thread 31936): #0 0x00000038742d44b8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aafd6c05c28 in epoll_dispatch (base=0xbbf8910, tv=) at epoll.c:404 #2 0x00002aafd6bf4a4c in event_base_loop (base=0xbbf8910, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xbbf7c10) at daemon/thread.c:304 #4 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 6 (Thread 31937): #0 0x00000038742d44b8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aafd6c05c28 in epoll_dispatch (base=0xbbf91b0, tv=) at epoll.c:404 #2 0x00002aafd6bf4a4c in event_base_loop (base=0xbbf91b0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xbbf7d10) at daemon/thread.c:304 #4 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 5 (Thread 31938): #0 0x00000038742d44b8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aafd6c05c28 in epoll_dispatch (base=0xbbf9a50, tv=) at epoll.c:404 #2 0x00002aafd6bf4a4c in event_base_loop (base=0xbbf9a50, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xbbf7e10) at daemon/thread.c:304 #4 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 4 (Thread 31939): #0 0x00000038742d44b8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002aafd6c05c28 in epoll_dispatch (base=0xbbfa3f0, tv=) at epoll.c:404 #2 0x00002aafd6bf4a4c in event_base_loop (base=0xbbfa3f0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0xbbf7f10) at daemon/thread.c:304 #4 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 3 (Thread 327): #0 0x0000003874e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaaaae71f in engine_shutdown_thread (arg=0xbc74bb0) at bucket_engine.c:1134 #2 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #3 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 2 (Thread 1304): #0 0x00000038742dfe8e in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00000038742770bd in _L_lock_5478 () from /lib64/libc.so.6 #2 0x0000003874271de3 in _int_free () from /lib64/libc.so.6 #3 0x00000038742728bb in free () from /lib64/libc.so.6 #4 0x00002aaaaad9ffaf in StoredValue::operator delete (p=0x15f5e760) at stored-value.hh:72 #5 0x00002aaaaadebb43 in HashTable::clear (this=0xf6b8c28, deactivate=true) at stored-value.cc:134 #6 0x00002aaaaad8afae in HashTable::~HashTable (this=0xf6b8c28, __in_chrg=) at stored-value.hh:845 #7 0x00002aaaaad8b6b1 in VBucket::~VBucket (this=0xf6b8c20, __in_chrg=) at vbucket.hh:115 #8 0x00002aaaaad8b8b3 in RCPtr::~RCPtr (this=0x2aaab4f7a558, __in_chrg=) at atomic.hh:340 #9 0x00002aaaaada2b9c in VBucketHolder::~VBucketHolder (this=0x2aaab438f140, __in_chrg=) at vbucket.hh:251 #10 0x00002aaaaada2dd3 in RCPtr::~RCPtr (this=0x4e76e060, __in_chrg=) at atomic.hh:340 #11 0x00002aaaaada2e61 in VBucketMap::getBucket (this=0x2aaab438ef50, id=943) at vbucket.hh:431 #12 0x00002aaaaada323f in EventuallyPersistentStore::visit ( this=0x2aaab438eee0, visitor=...) at ep.hh:655 #13 0x00002aaaaadb7908 in launch_backfill_thread (arg=0x2aaab1bca970) at ep_engine.cc:2481 #14 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #15 0x00000038742d40cd in clone () from /lib64/libc.so.6 Thread 1 (Thread 1314): #0 0x0000003874230265 in raise () from /lib64/libc.so.6 #1 0x0000003874231d10 in abort () from /lib64/libc.so.6 #2 0x000000387426a99b in __libc_message () from /lib64/libc.so.6 #3 0x0000003874272555 in _int_free () from /lib64/libc.so.6 #4 0x00000038742728bb in free () from /lib64/libc.so.6 #5 0x00002aaaaad9ffaf in StoredValue::operator delete (p=0x1346a260) at stored-value.hh:72 #6 0x00002aaaaadebb43 in HashTable::clear (this=0xf6b9238, deactivate=true) at stored-value.cc:134 #7 0x00002aaaaad8afae in HashTable::~HashTable (this=0xf6b9238, __in_chrg=) at stored-value.hh:845 #8 0x00002aaaaad8b6b1 in VBucket::~VBucket (this=0xf6b9230, __in_chrg=) at vbucket.hh:115 #9 0x00002aaaaad8b8b3 in RCPtr::~RCPtr (this=0x2aaab4f7a568, __in_chrg=) at atomic.hh:340 #10 0x00002aaaaada2b9c in VBucketHolder::~VBucketHolder (this=0x2aaab438f140, __in_chrg=) at vbucket.hh:251 #11 0x00002aaaaada2dd3 in RCPtr::~RCPtr (this=0x2aaab438ef50, __in_chrg=) at atomic.hh:340 #12 0x00002aaaaada309b in VBucketMap::~VBucketMap (this=0x2aaab438ef50, __in_chrg=) at vbucket.hh:402 #13 0x00002aaaaad99f38 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x2aaab438eee0, __in_chrg=) at ep.cc:426 #14 0x00002aaaaaddc3d0 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x2aaab421bfd0, __in_chrg=) at ep_engine.h:505 #15 0x00002aaaaadbe1d3 in EvpDestroy (handle=0x2aaab421bfd0, force=true) at ep_engine.cc:97 #16 0x00002aaaaaaae4ff in engine_shutdown_thread (arg=0x2aaab43bd260) at bucket_engine.c:1096 #17 0x0000003874e0673d in start_thread () from /lib64/libpthread.so.0 #18 0x00000038742d40cd in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 May 5 07:24 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 c7fc13e0591be3155fea1efcd88c0430 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 May 5 07:24 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 62b1f1fede447a7642ca0f09d84eebe6 /opt/membase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 12 Apr 13 15:11 /lib64/libdl.so.2 -> libdl-2.5.so a720040b8159c8a28106761465fd75f6 /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 11 Apr 13 15:11 /lib64/libm.so.6 -> libm-2.5.so e6caaa56b9a85609bab7350043ebf0ff /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 12 Apr 13 15:11 /lib64/librt.so.1 -> librt-2.5.so aa0c8b78aa0afa0e8b4bf571ed290036 /lib64/librt.so.1 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 17 Apr 13 15:11 /lib64/libpthread.so.0 -> libpthread-2.5.so 08ab2e8c0ea80365aea5d30718ec7110 /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 11 Apr 13 15:11 /lib64/libc.so.6 -> libc-2.5.so b6656e16db00f852c39ffff312118062 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 9 Apr 13 15:11 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so 4a8689c13a38901a2fe3f1f030ee4268 /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15322 May 5 02:48 /opt/membase/lib/memcached/stdin_term_handler.so 2b46627c47610208b90a34190ab97bb8 /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 May 5 07:24 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 3172eba599233ba7534d598e65afa23b /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 May 5 07:24 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 1.6.5.3_226_gca18338 EP engine v1.6.5.3_226_gca18338 21acdd1bdec6e88f6e02de984c337a3d /opt/membase/lib/memcached/ep.so /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 18 Apr 13 15:14 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.8 62cdd65712aafeaeef91bac91dfa67e1 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Apr 13 15:11 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1 fdb71a946f6f4ca92e99a9516ea10082 /lib64/libgcc_s.so.1