Basic crash dump analysis of /tmp/core.memcached.27383. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 membase membase 337460 2011-05-07 20:43 /opt/membase/bin/memcached -rw------- 1 membase membase 442957824 2011-05-08 07:15 /tmp/core.memcached.27383 4b89880b515caedfedcea1de302d7e7b /opt/membase/bin/memcached 320bbb7002a16561137cd873d70596dd /tmp/core.memcached.27383 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.0-ubuntu Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 30030] [New Thread 27388] [New Thread 27389] [New Thread 27390] [New Thread 27383] [New Thread 27384] [New Thread 27385] [New Thread 27386] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib/tls/i686/cmov/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libdl.so.2 Reading symbols from /lib/tls/i686/cmov/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libm.so.6 Reading symbols from /lib/tls/i686/cmov/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/librt.so.1 Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0 Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0xb77a7430 in __kernel_vsyscall () Thread 9 (Thread 27386): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7699286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb77843c5 in epoll_dispatch (base=0x8764320, tv=0x0) at epoll.c:404 #3 0xb7771894 in event_base_loop (base=0x8764320, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x8764020) at daemon/thread.c:304 #5 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 8 (Thread 27385): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7662b76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7662991 in sleep () from /lib/tls/i686/cmov/libc.so.6 #3 0x0805eeb0 in check_isasl_db_thread (arg=0x806aa38) at daemon/isasl.c:233 #4 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #5 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 7 (Thread 27384): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb76890bb in read () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7632fbb in _IO_file_underflow () from /lib/tls/i686/cmov/libc.so.6 #3 0xb763484b in _IO_default_uflow () from /lib/tls/i686/cmov/libc.so.6 #4 0xb7635c78 in __uflow () from /lib/tls/i686/cmov/libc.so.6 #5 0xb762b56c in getc () from /lib/tls/i686/cmov/libc.so.6 #6 0xb775f7ac in check_stdin_thread (arg=0x804ae70) at extensions/daemon/stdin_check.c:19 #7 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 6 (Thread 27383): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7699286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb77843c5 in epoll_dispatch (base=0x87596a8, tv=0xbfc376a0) at epoll.c:404 #3 0xb7771894 in event_base_loop (base=0x87596a8, flags=) at event.c:1558 #4 0x0805284c in main (argc=15, argv=0xbfc39174) at daemon/memcached.c:7459 Thread 5 (Thread 27390): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7699286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb77843c5 in epoll_dispatch (base=0x8765940, tv=0x0) at epoll.c:404 #3 0xb7771894 in event_base_loop (base=0x8765940, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x8764260) at daemon/thread.c:304 #5 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 4 (Thread 27389): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7699286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb77843c5 in epoll_dispatch (base=0x8765358, tv=0x0) at epoll.c:404 #3 0xb7771894 in event_base_loop (base=0x8765358, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x87641d0) at daemon/thread.c:304 #5 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 3 (Thread 27388): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7699286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb77843c5 in epoll_dispatch (base=0x8764df0, tv=0x0) at epoll.c:404 #3 0xb7771894 in event_base_loop (base=0x8764df0, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x8764140) at daemon/thread.c:304 #5 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 2 (Thread 30030): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb7694fb6 in munmap () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7638503 in ?? () from /lib/tls/i686/cmov/libc.so.6 #3 0xb763b8ad in free () from /lib/tls/i686/cmov/libc.so.6 #4 0x0805dbc2 in topkeys_free (tk=0xb1200010) at daemon/topkeys.c:69 #5 0x0804c6c0 in release_independent_stats (stats=0x874f110) at daemon/memcached.c:6325 #6 0xb65c08e6 in uninit_engine_handle (peh=0x87b2f48) at bucket_engine.c:577 #7 0xb65c091e in free_engine_handle (peh=0x0) at bucket_engine.c:585 #8 0xb65c0dbd in engine_shutdown_thread (arg=0x87b2f48) at bucket_engine.c:1162 #9 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #10 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 1 (Thread 27387): #0 0xb77a7430 in __kernel_vsyscall () #1 0xb75f64d1 in raise () from /lib/tls/i686/cmov/libc.so.6 #2 0xb75f9932 in abort () from /lib/tls/i686/cmov/libc.so.6 #3 0xb75ef648 in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 #4 0xb65c055e in must_lock (mutex=) at bucket_engine.c:305 #5 0xb65c1173 in bucket_get_stats_struct (handle=0xb65c70e0, cookie=0x967eef0) at bucket_engine.c:1413 #6 0x080583e7 in get_independent_stats (c=0x967eef0) at daemon/memcached.c:6334 #7 get_thread_stats (c=0x967eef0) at daemon/memcached.c:6344 #8 transmit (c=0x967eef0) at daemon/memcached.c:4978 #9 conn_mwrite (c=0x967eef0) at daemon/memcached.c:5375 #10 0x0805c55e in thread_libevent_process (fd=15, which=2, arg=0x87640b0) at daemon/thread.c:381 #11 0xb77719af in event_process_active_single_queue (base=0x8764888, flags=) at event.c:1308 #12 event_process_active (base=0x8764888, flags=) at event.c:1375 #13 event_base_loop (base=0x8764888, flags=) at event.c:1572 #14 0x0805cec7 in worker_libevent (arg=0x87640b0) at daemon/thread.c:304 #15 0xb771680e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #16 0xb7698a0e in clone () from /lib/tls/i686/cmov/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 membase membase 31 2011-05-07 20:59 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 18eeb43dde109e69e46dcf87006ffbc6 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 membase membase 21 2011-05-07 20:59 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 c24245821034d9036c5d5ec08843ffce /opt/membase/lib/libevent-2.0.so.5 /lib/tls/i686/cmov/libdl.so.2: lrwxrwxrwx 1 root root 15 2010-05-28 15:14 /lib/tls/i686/cmov/libdl.so.2 -> libdl-2.10.1.so 0c7d908be4733ca4ad389efa978d8a29 /lib/tls/i686/cmov/libdl.so.2 /lib/tls/i686/cmov/libm.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:14 /lib/tls/i686/cmov/libm.so.6 -> libm-2.10.1.so 0faddf5618f03f10299d1acc619cfa66 /lib/tls/i686/cmov/libm.so.6 /lib/tls/i686/cmov/librt.so.1: lrwxrwxrwx 1 root root 15 2010-05-28 15:14 /lib/tls/i686/cmov/librt.so.1 -> librt-2.10.1.so cedbebc23fda1c1f1bef052420432443 /lib/tls/i686/cmov/librt.so.1 /lib/tls/i686/cmov/libpthread.so.0: lrwxrwxrwx 1 root root 20 2010-05-28 15:14 /lib/tls/i686/cmov/libpthread.so.0 -> libpthread-2.10.1.so f229b04170f2a41f4662944ee618fd11 /lib/tls/i686/cmov/libpthread.so.0 /lib/tls/i686/cmov/libc.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:14 /lib/tls/i686/cmov/libc.so.6 -> libc-2.10.1.so ac51d130ca2be9d839aca53f1e7d74ab /lib/tls/i686/cmov/libc.so.6 /lib/ld-linux.so.2: lrwxrwxrwx 1 root root 12 2010-05-28 15:14 /lib/ld-linux.so.2 -> ld-2.10.1.so f4dd30e16c53f4e6d7d5e3e98a24d1b2 /lib/ld-linux.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rw-r--r-- 1 membase membase 15399 2011-05-07 20:43 /opt/membase/lib/memcached/stdin_term_handler.so 44da14a67de598f4e0e3689ac3c13e4a /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 membase membase 22 2011-05-07 20:59 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 e4a513b61a29410706acb259eeecee4f /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 membase membase 11 2011-05-07 20:59 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.6.5.3_239_g4643aef 1.6.5.3_239_g4643aef 1324156217b8eadd1f36ada3c244cd01 /opt/membase/lib/memcached/ep.so /usr/lib/libstdc++.so.6: lrwxrwxrwx 1 root root 19 2010-05-28 15:14 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.13 56896d5884e6c9bf1eecaa61e1ff6be4 /usr/lib/libstdc++.so.6 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 116272 2010-01-10 07:50 /lib/libgcc_s.so.1 69e882b26484cef67997f0895e99508a /lib/libgcc_s.so.1