Basic crash dump analysis of /tmp/core.memcached.20386. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 membase membase 337460 2011-05-07 20:43 /opt/membase/bin/memcached -rw------- 1 membase membase 489455616 2011-05-08 07:15 /tmp/core.memcached.20386 4b89880b515caedfedcea1de302d7e7b /opt/membase/bin/memcached 8cf2fc9f59d929e5a0332a2a5b5a9051 /tmp/core.memcached.20386 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.0-ubuntu Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 20408] [New Thread 20409] [New Thread 20410] [New Thread 20386] [New Thread 20387] [New Thread 20388] [New Thread 20389] [New Thread 20390] [New Thread 20391] [New Thread 20392] [New Thread 20393] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib/tls/i686/cmov/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libdl.so.2 Reading symbols from /lib/tls/i686/cmov/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libm.so.6 Reading symbols from /lib/tls/i686/cmov/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/librt.so.1 Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0 Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0xb7894430 in __kernel_vsyscall () Thread 12 (Thread 20393): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7786286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78713c5 in epoll_dispatch (base=0x9533940, tv=0x0) at epoll.c:404 #3 0xb785e894 in event_base_loop (base=0x9533940, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x9532260) at daemon/thread.c:304 #5 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 11 (Thread 20392): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7786286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78713c5 in epoll_dispatch (base=0x9533358, tv=0x0) at epoll.c:404 #3 0xb785e894 in event_base_loop (base=0x9533358, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x95321d0) at daemon/thread.c:304 #5 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 10 (Thread 20391): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7786286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78713c5 in epoll_dispatch (base=0x9532df0, tv=0x0) at epoll.c:404 #3 0xb785e894 in event_base_loop (base=0x9532df0, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x9532140) at daemon/thread.c:304 #5 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 9 (Thread 20390): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7786286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78713c5 in epoll_dispatch (base=0x9532888, tv=0x0) at epoll.c:404 #3 0xb785e894 in event_base_loop (base=0x9532888, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x95320b0) at daemon/thread.c:304 #5 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 8 (Thread 20389): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7786286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78713c5 in epoll_dispatch (base=0x9532320, tv=0x0) at epoll.c:404 #3 0xb785e894 in event_base_loop (base=0x9532320, flags=) at event.c:1558 #4 0x0805cec7 in worker_libevent (arg=0x9532020) at daemon/thread.c:304 #5 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 7 (Thread 20388): #0 0xb7894430 in __kernel_vsyscall () #1 0xb774fb76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6 #2 0xb774f991 in sleep () from /lib/tls/i686/cmov/libc.so.6 #3 0x0805eeb0 in check_isasl_db_thread (arg=0x806aa38) at daemon/isasl.c:233 #4 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #5 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 6 (Thread 20387): #0 0xb7894430 in __kernel_vsyscall () #1 0xb77760bb in read () from /lib/tls/i686/cmov/libc.so.6 #2 0xb771ffbb in _IO_file_underflow () from /lib/tls/i686/cmov/libc.so.6 #3 0xb772184b in _IO_default_uflow () from /lib/tls/i686/cmov/libc.so.6 #4 0xb7722c78 in __uflow () from /lib/tls/i686/cmov/libc.so.6 #5 0xb771856c in getc () from /lib/tls/i686/cmov/libc.so.6 #6 0xb784c7ac in check_stdin_thread (arg=0x804ae70) at extensions/daemon/stdin_check.c:19 #7 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #8 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 5 (Thread 20386): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7786286 in epoll_wait () from /lib/tls/i686/cmov/libc.so.6 #2 0xb78713c5 in epoll_dispatch (base=0x95276a8, tv=0xbfae2e30) at epoll.c:404 #3 0xb785e894 in event_base_loop (base=0x95276a8, flags=) at event.c:1558 #4 0x0805284c in main (argc=15, argv=0xbfae4904) at daemon/memcached.c:7459 Thread 4 (Thread 20410): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7808142 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb3d994f1 in SyncObject::wait (this=0xaffeb28, d=...) at syncobject.hh:42 #3 IdleTask::run (this=0xaffeb28, d=...) at dispatcher.cc:244 #4 0xb3d9b5a0 in Dispatcher::run (this=0xaffdfe8) at dispatcher.cc:119 #5 0xb3d9c395 in launch_dispatcher_thread (arg=0xaffdfe8) at dispatcher.cc:28 #6 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 3 (Thread 20409): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7792d03 in ?? () from /lib/tls/i686/cmov/libc.so.6 #2 0xb772a037 in ?? () from /lib/tls/i686/cmov/libc.so.6 Backtrace stopped: previous frame identical to this frame (corrupt stack?) Thread 2 (Thread 20408): #0 0xb7894430 in __kernel_vsyscall () #1 0xb7807e15 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb3d9b814 in SyncObject::wait (this=0xafccea8) at syncobject.hh:31 #3 Dispatcher::run (this=0xafccea8) at dispatcher.cc:85 #4 0xb3d9c395 in launch_dispatcher_thread (arg=0xafccea8) at dispatcher.cc:28 #5 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 1 (Thread 23034): #0 0xb7894430 in __kernel_vsyscall () #1 0xb76e34d1 in raise () from /lib/tls/i686/cmov/libc.so.6 #2 0xb76e6932 in abort () from /lib/tls/i686/cmov/libc.so.6 #3 0xb76dc648 in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 #4 0xb66ae464 in bucket_engine_release_cookie (cookie=0xb1e0688) at bucket_engine.c:1989 #5 0xb3df0592 in TapConnection::releaseReference (this=0xb04e198, force=false) at tapconnection.cc:31 #6 0xb3e00d90 in TapConnectionReaperCallback (this=0x95d45bc) at tapconnmap.cc:21 #7 TapConnMap::shutdownAllTapConnections (this=0x95d45bc) at tapconnmap.cc:341 #8 0xb3dbc66c in EventuallyPersistentEngine::destroy (this=0x95d42e0, force=true) at ep_engine.cc:1438 #9 0xb3dc5597 in EvpDestroy (handle=0x95d42e0, force=) at ep_engine.cc:96 #10 0xb66adbac in engine_shutdown_thread (arg=0x9580f48) at bucket_engine.c:1096 #11 0xb780380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #12 0xb7785a0e in clone () from /lib/tls/i686/cmov/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 membase membase 31 2011-05-07 21:01 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 18eeb43dde109e69e46dcf87006ffbc6 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 membase membase 21 2011-05-07 21:01 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 c24245821034d9036c5d5ec08843ffce /opt/membase/lib/libevent-2.0.so.5 /lib/tls/i686/cmov/libdl.so.2: lrwxrwxrwx 1 root root 15 2010-05-28 15:14 /lib/tls/i686/cmov/libdl.so.2 -> libdl-2.10.1.so 0c7d908be4733ca4ad389efa978d8a29 /lib/tls/i686/cmov/libdl.so.2 /lib/tls/i686/cmov/libm.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:14 /lib/tls/i686/cmov/libm.so.6 -> libm-2.10.1.so 0faddf5618f03f10299d1acc619cfa66 /lib/tls/i686/cmov/libm.so.6 /lib/tls/i686/cmov/librt.so.1: lrwxrwxrwx 1 root root 15 2010-05-28 15:14 /lib/tls/i686/cmov/librt.so.1 -> librt-2.10.1.so cedbebc23fda1c1f1bef052420432443 /lib/tls/i686/cmov/librt.so.1 /lib/tls/i686/cmov/libpthread.so.0: lrwxrwxrwx 1 root root 20 2010-05-28 15:14 /lib/tls/i686/cmov/libpthread.so.0 -> libpthread-2.10.1.so f229b04170f2a41f4662944ee618fd11 /lib/tls/i686/cmov/libpthread.so.0 /lib/tls/i686/cmov/libc.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:14 /lib/tls/i686/cmov/libc.so.6 -> libc-2.10.1.so ac51d130ca2be9d839aca53f1e7d74ab /lib/tls/i686/cmov/libc.so.6 /lib/ld-linux.so.2: lrwxrwxrwx 1 root root 12 2010-05-28 15:14 /lib/ld-linux.so.2 -> ld-2.10.1.so f4dd30e16c53f4e6d7d5e3e98a24d1b2 /lib/ld-linux.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rw-r--r-- 1 membase membase 15399 2011-05-07 20:43 /opt/membase/lib/memcached/stdin_term_handler.so 44da14a67de598f4e0e3689ac3c13e4a /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 membase membase 22 2011-05-07 21:01 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 e4a513b61a29410706acb259eeecee4f /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 membase membase 11 2011-05-07 21:01 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.6.5.3_239_g4643aef 1.6.5.3_239_g4643aef 1324156217b8eadd1f36ada3c244cd01 /opt/membase/lib/memcached/ep.so /usr/lib/libstdc++.so.6: lrwxrwxrwx 1 root root 19 2010-05-28 15:14 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.13 56896d5884e6c9bf1eecaa61e1ff6be4 /usr/lib/libstdc++.so.6 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 116272 2010-01-10 07:50 /lib/libgcc_s.so.1 69e882b26484cef67997f0895e99508a /lib/libgcc_s.so.1