Basic crash dump analysis of /tmp/core.memcached.18635. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 membase membase 337460 2011-05-07 20:43 /opt/membase/bin/memcached -rw------- 1 membase membase 455651328 2011-05-08 07:16 /tmp/core.memcached.18635 4b89880b515caedfedcea1de302d7e7b /opt/membase/bin/memcached 1b4938917b931cd9de5361d2fc9f6cbe /tmp/core.memcached.18635 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.0-ubuntu Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 18656] [New Thread 18658] [New Thread 18635] [New Thread 18637] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib/tls/i686/cmov/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libdl.so.2 Reading symbols from /lib/tls/i686/cmov/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libm.so.6 Reading symbols from /lib/tls/i686/cmov/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/librt.so.1 Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0 Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0xb774d430 in __kernel_vsyscall () Thread 5 (Thread 18637): #0 0xb774d430 in __kernel_vsyscall () #1 0xb7608b76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6 #2 0xb7608991 in sleep () from /lib/tls/i686/cmov/libc.so.6 #3 0x0805eeb0 in check_isasl_db_thread (arg=0x806aa38) at daemon/isasl.c:233 #4 0xb76bc80e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #5 0xb763ea0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 4 (Thread 18635): #0 0xb774d430 in __kernel_vsyscall () #1 0xb7608b76 in nanosleep () from /lib/tls/i686/cmov/libc.so.6 #2 0xb76384cc in usleep () from /lib/tls/i686/cmov/libc.so.6 #3 0xb3c9242c in Flusher::wait (this=0xae74ba0) at flusher.cc:33 #4 0xb3c576ab in EventuallyPersistentStore::stopFlusher (this=0xa1b26f8) at ep.cc:447 #5 0xb3c5805a in ~EventuallyPersistentStore (this=0xa1b26f8, __in_chrg=) at ep.cc:414 #6 0xb3c7e5bf in ~EventuallyPersistentEngine (handle=0x944a2e0, force=) at ep_engine.h:505 #7 EvpDestroy (handle=0x944a2e0, force=) at ep_engine.cc:97 #8 0xb65664da in bucket_shutdown_engine (key=0x944fee0, nkey=53, val=0x93f6f48, nval=0, args=0x0) at bucket_engine.c:1027 #9 0xb77474ac in genhash_iter (h=0x939dda8, iterfunc=0xb6566480 , arg=0x0) at utilities/genhash.c:270 #10 0xb6568eec in bucket_destroy (handle=0xb656d0e0, force=false) at bucket_engine.c:1051 #11 0x0805288d in main (argc=15, argv=0xbfebdee4) at daemon/memcached.c:7467 Thread 3 (Thread 18658): #0 0xb774d430 in __kernel_vsyscall () #1 0xb76c1142 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb3c524f1 in SyncObject::wait (this=0xae74b28, d=...) at syncobject.hh:42 #3 IdleTask::run (this=0xae74b28, d=...) at dispatcher.cc:244 #4 0xb3c545a0 in Dispatcher::run (this=0xae73fe8) at dispatcher.cc:119 #5 0xb3c55395 in launch_dispatcher_thread (arg=0xae73fe8) at dispatcher.cc:28 #6 0xb76bc80e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #7 0xb763ea0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 2 (Thread 18656): #0 0xb774d430 in __kernel_vsyscall () #1 0xb76c0e15 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/i686/cmov/libpthread.so.0 #2 0xb3c54814 in SyncObject::wait (this=0xae42ea8) at syncobject.hh:31 #3 Dispatcher::run (this=0xae42ea8) at dispatcher.cc:85 #4 0xb3c55395 in launch_dispatcher_thread (arg=0xae42ea8) at dispatcher.cc:28 #5 0xb76bc80e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #6 0xb763ea0e in clone () from /lib/tls/i686/cmov/libc.so.6 Thread 1 (Thread 18657): #0 0xb774d430 in __kernel_vsyscall () #1 0xb759c4d1 in raise () from /lib/tls/i686/cmov/libc.so.6 #2 0xb759f932 in abort () from /lib/tls/i686/cmov/libc.so.6 #3 0xb7595648 in __assert_fail () from /lib/tls/i686/cmov/libc.so.6 #4 0xb3c93736 in Flusher::step (this=0xae74ba0, d=..., tid=...) at flusher.cc:190 #5 0xb3c942db in FlusherStepper::callback (this=0xb1900900, d=..., t=...) at flusher.cc:23 #6 0xb3c55b79 in Task::run (this=0xb1900928, d=..., t=...) at dispatcher.hh:139 #7 0xb3c545a0 in Dispatcher::run (this=0xa1f94b8) at dispatcher.cc:119 #8 0xb3c55395 in launch_dispatcher_thread (arg=0xa1f94b8) at dispatcher.cc:28 #9 0xb76bc80e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0 #10 0xb763ea0e in clone () from /lib/tls/i686/cmov/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 membase membase 31 2011-05-07 21:01 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 18eeb43dde109e69e46dcf87006ffbc6 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 membase membase 21 2011-05-07 21:01 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 c24245821034d9036c5d5ec08843ffce /opt/membase/lib/libevent-2.0.so.5 /lib/tls/i686/cmov/libdl.so.2: lrwxrwxrwx 1 root root 15 2010-05-28 15:14 /lib/tls/i686/cmov/libdl.so.2 -> libdl-2.10.1.so 0c7d908be4733ca4ad389efa978d8a29 /lib/tls/i686/cmov/libdl.so.2 /lib/tls/i686/cmov/libm.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:14 /lib/tls/i686/cmov/libm.so.6 -> libm-2.10.1.so 0faddf5618f03f10299d1acc619cfa66 /lib/tls/i686/cmov/libm.so.6 /lib/tls/i686/cmov/librt.so.1: lrwxrwxrwx 1 root root 15 2010-05-28 15:14 /lib/tls/i686/cmov/librt.so.1 -> librt-2.10.1.so cedbebc23fda1c1f1bef052420432443 /lib/tls/i686/cmov/librt.so.1 /lib/tls/i686/cmov/libpthread.so.0: lrwxrwxrwx 1 root root 20 2010-05-28 15:14 /lib/tls/i686/cmov/libpthread.so.0 -> libpthread-2.10.1.so f229b04170f2a41f4662944ee618fd11 /lib/tls/i686/cmov/libpthread.so.0 /lib/tls/i686/cmov/libc.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:14 /lib/tls/i686/cmov/libc.so.6 -> libc-2.10.1.so ac51d130ca2be9d839aca53f1e7d74ab /lib/tls/i686/cmov/libc.so.6 /lib/ld-linux.so.2: lrwxrwxrwx 1 root root 12 2010-05-28 15:14 /lib/ld-linux.so.2 -> ld-2.10.1.so f4dd30e16c53f4e6d7d5e3e98a24d1b2 /lib/ld-linux.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rw-r--r-- 1 membase membase 15399 2011-05-07 20:43 /opt/membase/lib/memcached/stdin_term_handler.so 44da14a67de598f4e0e3689ac3c13e4a /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 membase membase 22 2011-05-07 21:01 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 e4a513b61a29410706acb259eeecee4f /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 membase membase 11 2011-05-07 21:01 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.6.5.3_239_g4643aef 1.6.5.3_239_g4643aef 1324156217b8eadd1f36ada3c244cd01 /opt/membase/lib/memcached/ep.so /usr/lib/libstdc++.so.6: lrwxrwxrwx 1 root root 19 2010-05-28 15:14 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.13 56896d5884e6c9bf1eecaa61e1ff6be4 /usr/lib/libstdc++.so.6 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 116272 2010-01-10 07:50 /lib/libgcc_s.so.1 69e882b26484cef67997f0895e99508a /lib/libgcc_s.so.1