Basic crash dump analysis of /tmp//core.memcached.21850. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 membase membase 412558 2011-05-15 00:25 /opt/membase/bin/memcached -rw------- 1 membase membase 680247296 2011-05-15 07:01 /tmp//core.memcached.21850 65cd74046aa972bdb28529ba0bf1190a /opt/membase/bin/memcached f9eb9c9db3346b42ebe194032b7f32eb /tmp//core.memcached.21850 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 22470] [New Thread 21850] [New Thread 21855] [New Thread 21857] [New Thread 21858] [New Thread 21859] [New Thread 21860] [New Thread 21861] [New Thread 21874] [New Thread 21876] [New Thread 21856] [New Thread 21875] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x00007f0347746a75 in raise () from /lib/libc.so.6 Thread 12 (Thread 21875): #0 0x00007f03477f231d in fdatasync () from /lib/libc.so.6 #1 0x00007f0343830119 in full_fsync (id=0x7f03385df4d0, flags=2) at embedded/sqlite3.c:25510 #2 unixSync (id=0x7f03385df4d0, flags=2) at embedded/sqlite3.c:25558 #3 0x00007f0343850503 in sqlite3OsSync (pWal=0x7f03385df458, szPage=1024, pList=0x7f0331ad5ca8, nTruncate=41602, isCommit=1, sync_flags=2) at embedded/sqlite3.c:13413 #4 sqlite3WalFrames (pWal=0x7f03385df458, szPage=1024, pList=0x7f0331ad5ca8, nTruncate=41602, isCommit=1, sync_flags=2) at embedded/sqlite3.c:43360 #5 0x00007f03438605c5 in pagerWalFrames (pPager=0x7f0331aa34b8, zMaster=, noSync=0) at embedded/sqlite3.c:37082 #6 sqlite3PagerCommitPhaseOne (pPager=0x7f0331aa34b8, zMaster=, noSync=0) at embedded/sqlite3.c:39803 #7 0x00007f034387b487 in sqlite3BtreeCommitPhaseOne (p=0x7f0331a64b18) at embedded/sqlite3.c:47605 #8 vdbeCommit (p=0x7f0331a64b18) at embedded/sqlite3.c:56090 #9 sqlite3VdbeHalt (p=0x7f0331a64b18) at embedded/sqlite3.c:56514 #10 0x00007f03438a1271 in sqlite3VdbeExec (p=0x7f0331a64b18) at embedded/sqlite3.c:60326 #11 0x00007f03438a27dd in sqlite3Step (pStmt=) at embedded/sqlite3.c:57947 #12 sqlite3_step (pStmt=) at embedded/sqlite3.c:58011 #13 0x00007f0343820faa in PreparedStatement::execute (this=0x7f0342a672e0) at sqlite-pst.cc:73 #14 0x00007f0343822b68 in SqliteStrategy::execute ( this=, query=) at sqlite-strategies.cc:151 #15 0x00007f034382310b in ShardedByVBucketSqliteStrategy::destroyTables ( this=0x1b50420) at sqlite-strategies.cc:417 #16 0x00007f034381f5af in StrategicSqlite3::reset (this=0x1b504e0) at sqlite-kvstore.cc:145 #17 0x00007f03437c71cd in EventuallyPersistentStore::flushOneDeleteAll ( this=0x2b67820, q=0x2b67940, rejectQueue=0x7f0331ad6cb0) at ep.cc:1731 #18 EventuallyPersistentStore::flushOne (this=0x2b67820, q=0x2b67940, rejectQueue=0x7f0331ad6cb0) at ep.cc:1869 #19 0x00007f03437c741a in EventuallyPersistentStore::flushSome ( this=0x2b67820, q=0x2b67940, rejectQueue=0x7f0331ad6cb0) at ep.cc:1487 #20 0x00007f03437f6cba in Flusher::doFlush (this=0x3be9920) at flusher.cc:240 #21 0x00007f03437f7715 in Flusher::step (this=0x36, d=..., tid=...) at flusher.cc:154 #22 0x00007f03437f7c4a in FlusherStepper::callback ( this=, d=..., t=) at flusher.cc:23 #23 0x00007f03437bae9a in Task::run (this=, d=..., t=) at dispatcher.hh:139 #24 0x00007f03437b9b82 in Dispatcher::run (this=0x2ca8e80) at dispatcher.cc:119 #25 0x00007f03437ba793 in launch_dispatcher_thread (arg=0x36) at dispatcher.cc:28 #26 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #27 0x00007f03477f96cd in clone () from /lib/libc.so.6 #28 0x0000000000000000 in ?? () Thread 11 (Thread 21856): #0 0x00007f03477bd35d in nanosleep () from /lib/libc.so.6 #1 0x00007f03477bd1d0 in sleep () from /lib/libc.so.6 #2 0x0000000000416618 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #4 0x00007f03477f96cd in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 10 (Thread 21876): #0 0x00007f0347aa0bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f03437b7cc8 in SyncObject::wait (this=0x3be9840, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x3be9840, d=...) at dispatcher.cc:244 #3 0x00007f03437b9b82 in Dispatcher::run (this=0x3be87b0) at dispatcher.cc:119 #4 0x00007f03437ba793 in launch_dispatcher_thread (arg=0x3be87fc) at dispatcher.cc:28 #5 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #6 0x00007f03477f96cd in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 9 (Thread 21874): #0 0x00007f0347aa085c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007f03437b9ded in SyncObject::wait (this=0x3b83e90) at syncobject.hh:31 #2 Dispatcher::run (this=0x3b83e90) at dispatcher.cc:85 #3 0x00007f03437ba793 in launch_dispatcher_thread (arg=0x3b83edc) at dispatcher.cc:28 #4 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f03477f96cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 8 (Thread 21861): #0 0x00007f03477f9cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007f034836495c in epoll_dispatch (base=0x1b5e4e0, tv=) at epoll.c:404 #2 0x00007f03483532cc in event_base_loop (base=0x1b5e4e0, flags=) at event.c:1558 #3 0x0000000000414824 in worker_libevent (arg=0x1b5c000) at daemon/thread.c:304 #4 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f03477f96cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 7 (Thread 21860): #0 0x00007f03477f9cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007f034836495c in epoll_dispatch (base=0x1b5db40, tv=) at epoll.c:404 #2 0x00007f03483532cc in event_base_loop (base=0x1b5db40, flags=) at event.c:1558 #3 0x0000000000414824 in worker_libevent (arg=0x1b5bf00) at daemon/thread.c:304 #4 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f03477f96cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (Thread 21859): #0 0x00007f03477f9cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007f034836495c in epoll_dispatch (base=0x1b5d2a0, tv=) at epoll.c:404 #2 0x00007f03483532cc in event_base_loop (base=0x1b5d2a0, flags=) at event.c:1558 #3 0x0000000000414824 in worker_libevent (arg=0x1b5be00) at daemon/thread.c:304 #4 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f03477f96cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 5 (Thread 21858): #0 0x00007f03477f9cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007f034836495c in epoll_dispatch (base=0x1b5ca00, tv=) at epoll.c:404 #2 0x00007f03483532cc in event_base_loop (base=0x1b5ca00, flags=) at event.c:1558 #3 0x0000000000414824 in worker_libevent (arg=0x1b5bd00) at daemon/thread.c:304 #4 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f03477f96cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (Thread 21857): #0 0x00007f03477f9cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007f034836495c in epoll_dispatch (base=0x1b5c160, tv=) at epoll.c:404 #2 0x00007f03483532cc in event_base_loop (base=0x1b5c160, flags=) at event.c:1558 #3 0x0000000000414824 in worker_libevent (arg=0x1b5bc00) at daemon/thread.c:304 #4 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #5 0x00007f03477f96cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 3 (Thread 21855): #0 0x00007f03477eb47d in read () from /lib/libc.so.6 #1 0x00007f0347786348 in _IO_file_underflow () from /lib/libc.so.6 #2 0x00007f0347787eee in _IO_default_uflow () from /lib/libc.so.6 #3 0x00007f034777ec7b in getc () from /lib/libc.so.6 #4 0x00007f0347511965 in check_stdin_thread (arg=0x403b70) at extensions/daemon/stdin_check.c:19 #5 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #6 0x00007f03477f96cd in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 2 (Thread 21850): #0 0x00007f03477f9cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007f034836495c in epoll_dispatch (base=0x1b50b40, tv=) at epoll.c:404 #2 0x00007f03483532cc in event_base_loop (base=0x1b50b40, flags=) at event.c:1558 #3 0x000000000040ae28 in main (argc=, argv=0x7fff8c794738) at daemon/memcached.c:7459 Thread 1 (Thread 22470): #0 0x00007f0347746a75 in raise () from /lib/libc.so.6 #1 0x00007f034774a5c0 in abort () from /lib/libc.so.6 #2 0x00007f034773f941 in __assert_fail () from /lib/libc.so.6 #3 0x00007f0346307bb8 in bucket_engine_release_cookie (cookie=0x3ea0da8) at bucket_engine.c:2004 #4 0x00007f034380a04f in TapConnection::releaseReference (this=0x3f24e20, force=198) at tapconnection.cc:35 #5 0x00007f034381b6d1 in TapConnectionReaperCallback::TapConnectionReaperCallback(EventuallyPersistentEngine&, TapConnection*) () from /opt/membase/lib/memcached/ep.so #6 0x00007f03438185da in TapConnMap::shutdownAllTapConnections ( this=0x1b47cc0) at tapconnmap.cc:341 #7 0x00007f03437da38a in EventuallyPersistentEngine::destroy ( this=0x1b47720, force=) at ep_engine.cc:1734 #8 0x00007f03437db539 in EvpDestroy (handle=0x1b47720, force=198) at ep_engine.cc:96 #9 0x00007f034630755d in engine_shutdown_thread (arg=0x1bd8cf0) at bucket_engine.c:1099 #10 0x00007f0347a9b9ca in start_thread () from /lib/libpthread.so.0 #11 0x00007f03477f96cd in clone () from /lib/libc.so.6 #12 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 membase membase 31 2011-05-15 02:52 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 685d6e2b61ffe61cad24b15050d44c05 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 membase membase 21 2011-05-15 02:52 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 ae6ec7b927064890ae7d820f067d8fca /opt/membase/lib/libevent-2.0.so.5 /lib/libdl.so.2: lrwxrwxrwx 1 root root 15 2010-05-28 15:10 /lib/libdl.so.2 -> libdl-2.11.1.so 4fc2038540e79999571aa4a338ac8692 /lib/libdl.so.2 /lib/libm.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:10 /lib/libm.so.6 -> libm-2.11.1.so 54f6f084ce7ce998d5ae8efaf5daea70 /lib/libm.so.6 /lib/librt.so.1: lrwxrwxrwx 1 root root 15 2010-05-28 15:10 /lib/librt.so.1 -> librt-2.11.1.so 1cfe1e6a4c1630a2e050303652c3341d /lib/librt.so.1 /lib/libpthread.so.0: lrwxrwxrwx 1 root root 20 2010-05-28 15:10 /lib/libpthread.so.0 -> libpthread-2.11.1.so deb7c9592f790b5ec2910b627df2d480 /lib/libpthread.so.0 /lib/libc.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:10 /lib/libc.so.6 -> libc-2.11.1.so 3a9c344619f5243066e844d540de6867 /lib/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 12 2010-05-28 15:10 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so 25dd1c7aa9cbf6535dc0ec73f6b94b62 /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rw-r--r-- 1 membase membase 17427 2011-05-15 00:25 /opt/membase/lib/memcached/stdin_term_handler.so 02e2ef2d3374e35e45965234c83f552c /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 membase membase 22 2011-05-15 02:52 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 926717ace0bd3f0a0dbbdf96ef275949 /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 membase membase 11 2011-05-15 02:52 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 1.6.5.3_269_g98e98f2 EP engine v1.6.5.3_269_g98e98f2 a97ccbec1dd9d6b5080804132d568621 /opt/membase/lib/memcached/ep.so /usr/lib/libstdc++.so.6: lrwxrwxrwx 1 root root 19 2010-05-28 12:00 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.13 145a441e95ff795ace078ba78840b91f /usr/lib/libstdc++.so.6 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 92552 2010-03-26 17:12 /lib/libgcc_s.so.1 2fb8dc6079174e77e399d0d5b57b11e5 /lib/libgcc_s.so.1