Basic crash dump analysis of /tmp/core.memcached.15486. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 324061 May 7 20:40 /opt/membase/./bin/memcached -rw------- 1 membase membase 431693824 May 7 23:05 /tmp/core.memcached.15486 a619ddb540f7de7821e404d0173ca9a8 /opt/membase/./bin/memcached 1bbc033ff5c5dd5dffa73614f4ae863d /tmp/core.memcached.15486 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 15493] [New Thread 15492] [New Thread 15491] [New Thread 15490] [New Thread 15488] [New Thread 15486] warning: .dynamic section for "/lib/libc.so.6" is not at the expected address warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x00c87402 in __kernel_vsyscall () Thread 7 (Thread 15486): #0 0x00c87402 in __kernel_vsyscall () #1 0x00299376 in epoll_wait () from /lib/libc.so.6 #2 0x001adb10 in epoll_dispatch (base=0x9ca9698, tv=0xbfbe6410) at epoll.c:404 #3 0x0019b0db in event_base_loop (base=0x9ca9698, flags=0) at event.c:1558 #4 0x08051449 in main (argc=15, argv=0xbfbe7ec4) at daemon/memcached.c:7459 Thread 6 (Thread 15488): #0 0x00c87402 in __kernel_vsyscall () #1 0x00258996 in nanosleep () from /lib/libc.so.6 #2 0x002587bf in sleep () from /lib/libc.so.6 #3 0x0805dc98 in check_isasl_db_thread (arg=0x8068b58) at daemon/isasl.c:233 #4 0x00b3e5ab in start_thread () from /lib/libpthread.so.0 #5 0x00298cfe in clone () from /lib/libc.so.6 Thread 5 (Thread 15490): #0 0x00c87402 in __kernel_vsyscall () #1 0x00299376 in epoll_wait () from /lib/libc.so.6 #2 0x001adb10 in epoll_dispatch (base=0x9cb4858, tv=0x0) at epoll.c:404 #3 0x0019b0db in event_base_loop (base=0x9cb4858, flags=0) at event.c:1558 #4 0x0805be97 in worker_libevent (arg=0x9cb4080) at daemon/thread.c:304 #5 0x00b3e5ab in start_thread () from /lib/libpthread.so.0 #6 0x00298cfe in clone () from /lib/libc.so.6 Thread 4 (Thread 15491): #0 0x00c87402 in __kernel_vsyscall () #1 0x00299376 in epoll_wait () from /lib/libc.so.6 #2 0x001adb10 in epoll_dispatch (base=0x9cb4dc0, tv=0x0) at epoll.c:404 #3 0x0019b0db in event_base_loop (base=0x9cb4dc0, flags=0) at event.c:1558 #4 0x0805be97 in worker_libevent (arg=0x9cb4110) at daemon/thread.c:304 #5 0x00b3e5ab in start_thread () from /lib/libpthread.so.0 #6 0x00298cfe in clone () from /lib/libc.so.6 Thread 3 (Thread 15492): #0 0x00c87402 in __kernel_vsyscall () #1 0x00299376 in epoll_wait () from /lib/libc.so.6 #2 0x001adb10 in epoll_dispatch (base=0x9cb5328, tv=0x0) at epoll.c:404 #3 0x0019b0db in event_base_loop (base=0x9cb5328, flags=0) at event.c:1558 #4 0x0805be97 in worker_libevent (arg=0x9cb41a0) at daemon/thread.c:304 #5 0x00b3e5ab in start_thread () from /lib/libpthread.so.0 #6 0x00298cfe in clone () from /lib/libc.so.6 Thread 2 (Thread 15493): #0 0x00c87402 in __kernel_vsyscall () #1 0x00299376 in epoll_wait () from /lib/libc.so.6 #2 0x001adb10 in epoll_dispatch (base=0x9cb5910, tv=0x0) at epoll.c:404 #3 0x0019b0db in event_base_loop (base=0x9cb5910, flags=0) at event.c:1558 #4 0x0805be97 in worker_libevent (arg=0x9cb4230) at daemon/thread.c:304 #5 0x00b3e5ab in start_thread () from /lib/libpthread.so.0 #6 0x00298cfe in clone () from /lib/libc.so.6 Thread 1 (Thread 15489): #0 0x00c87402 in __kernel_vsyscall () #1 0x001efdf0 in raise () from /lib/libc.so.6 #2 0x001f1701 in abort () from /lib/libc.so.6 #3 0x001e926b in __assert_fail () from /lib/libc.so.6 #4 0x004e24de in must_lock (mutex=) at bucket_engine.c:305 #5 0x004e2963 in bucket_get_stats_struct (handle=0x4e7200, cookie=0x9d26278) at bucket_engine.c:1413 #6 0x08055bd7 in transmit (c=0x9d26278) at daemon/memcached.c:6334 #7 conn_mwrite (c=0x9d26278) at daemon/memcached.c:5375 #8 0x08055ed5 in conn_write (c=0x9d26278) at daemon/memcached.c:5362 #9 0x0805b606 in thread_libevent_process (fd=10, which=2, arg=0x9cb3ff0) at daemon/thread.c:381 #10 0x0019b473 in event_process_active_single_queue (base=0x9cb42f0, flags=0) at event.c:1308 #11 event_process_active (base=0x9cb42f0, flags=0) at event.c:1375 #12 event_base_loop (base=0x9cb42f0, flags=0) at event.c:1572 #13 0x0805be97 in worker_libevent (arg=0x9cb3ff0) at daemon/thread.c:304 #14 0x00b3e5ab in start_thread () from /lib/libpthread.so.0 #15 0x00298cfe in clone () from /lib/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 May 7 21:00 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 d7f3cc38fec742b6369c184bcdeea869 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 May 7 21:00 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 41306044d1323b8eb2470f8a898c2d71 /opt/membase/lib/libevent-2.0.so.5 /lib/libdl.so.2: lrwxrwxrwx 1 root root 12 Jan 21 20:13 /lib/libdl.so.2 -> libdl-2.5.so 5595bab96891adb2388fd11508bade94 /lib/libdl.so.2 /lib/libm.so.6: lrwxrwxrwx 1 root root 11 Jan 21 20:13 /lib/libm.so.6 -> libm-2.5.so 39db4760139ab1dd2110c58a4c7dfd00 /lib/libm.so.6 /lib/librt.so.1: lrwxrwxrwx 1 root root 12 Jan 21 20:13 /lib/librt.so.1 -> librt-2.5.so 2b2b0f8d33912ac995de1dac02f10664 /lib/librt.so.1 /lib/libpthread.so.0: lrwxrwxrwx 1 root root 17 Jan 21 20:13 /lib/libpthread.so.0 -> libpthread-2.5.so 4f43284f07991dcb89b83beb93d25a71 /lib/libpthread.so.0 /lib/libc.so.6: lrwxrwxrwx 1 root root 11 Jan 21 20:13 /lib/libc.so.6 -> libc-2.5.so ee47dc8f004c2519ff2e33802fb6aed2 /lib/libc.so.6 /lib/ld-linux.so.2: lrwxrwxrwx 1 root root 9 Jan 21 20:13 /lib/ld-linux.so.2 -> ld-2.5.so 90f1806b4f80968626c213e0570c8247 /lib/ld-linux.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 12808 May 7 20:40 /opt/membase/lib/memcached/stdin_term_handler.so f36e64c8a23a1ce0107b8895bb9a1aa0 /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 May 7 21:00 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 4267fcc3f9c696276f5ac4465b69ee5d /opt/membase/lib/memcached/bucket_engine.so