Basic crash dump analysis of /tmp//core.memcached.4652. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 membase membase 412510 2011-05-06 19:48 /opt/membase/bin/memcached -rw------- 1 membase membase 491921408 2011-05-06 21:52 /tmp//core.memcached.4652 6eb0f2a2b8dd850ec36903690ddaccce /opt/membase/bin/memcached 743a4183f704d79f6c3afca197ec7e9c /tmp//core.memcached.4652 memcached 1.4.4_459_g6e709ec memcached 1.4.4_459_g6e709ec VERSION 1.4.4_459_g6e709ec -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 4736] [New Thread 4652] [New Thread 4653] [New Thread 4654] [New Thread 4656] [New Thread 4657] [New Thread 4733] [New Thread 4735] [New Thread 4655] [New Thread 4659] [New Thread 4734] [New Thread 4658] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x00007fbff149ea75 in raise () from /lib/libc.so.6 Thread 12 (Thread 4658): #0 0x00007fbff14f1d8b in memcpy () from /lib/libc.so.6 #1 0x00007fbff14e8d4b in ?? () from /lib/libc.so.6 #2 0x00007fbff14e9080 in realloc () from /lib/libc.so.6 #3 0x000000000040df75 in try_read_network (c=0x46b1ee8) at daemon/memcached.c:4863 #4 conn_read (c=0x46b1ee8) at daemon/memcached.c:5157 #5 0x000000000040724b in event_handler (fd=, which=, arg=0x46b1ee8) at daemon/memcached.c:5576 #6 0x00007fbff20ab3cd in event_process_active_single_queue (base=0x19b7b00, flags=) at event.c:1308 #7 event_process_active (base=0x19b7b00, flags=) at event.c:1375 #8 event_base_loop (base=0x19b7b00, flags=) at event.c:1572 #9 0x0000000000414824 in worker_libevent (arg=0x19b5ec0) at daemon/thread.c:304 #10 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #11 0x00007fbff15516cd in clone () from /lib/libc.so.6 #12 0x0000000000000000 in ?? () Thread 11 (Thread 4734): #0 LockHolder (this=0x7fbfec709620, from=...) at locks.hh:35 #1 0x00007fbfed4f9596 in HashTable::getLockedBucket (this=0x7fbfe40af3b8, h=-2141552478, bucket=0x7fbfec70966c) at stored-value.hh:1203 #2 0x00007fbfed4f961d in HashTable::getLockedBucket (this=0x7fbfe40af3b8, s=..., bucket=0x7fbfec70966c) at stored-value.hh:1230 #3 0x00007fbfed4fbc94 in PersistenceCallback::callback(std::pair&) () from /opt/membase/lib/memcached/ep.so #4 0x00007fbfed56138d in StrategicSqlite3::update (this=0x19aa3e0, itm=..., vb_version=5, cb=...) at sqlite-kvstore.cc:82 #5 0x00007fbfed56165b in StrategicSqlite3::set (this=0x19aa3e0, itm=..., vb_version=5, cb=...) at sqlite-kvstore.cc:110 #6 0x00007fbfed4e4ac2 in EventuallyPersistentStore::flushOneDelOrSet ( this=0x2aaf820, qi=..., rejectQueue=0x7fbfe45bcd50) at ep.cc:1828 #7 0x00007fbfed4e4eed in EventuallyPersistentStore::flushOne ( this=0x2aaf820, q=0x2aaf940, rejectQueue=0x7fbfe45bcd50) at ep.cc:1871 #8 0x00007fbfed4e50a9 in EventuallyPersistentStore::flushSome ( this=0x2aaf820, q=0x2aaf940, rejectQueue=0x7fbfe45bcd50) at ep.cc:1483 #9 0x00007fbfed52fd75 in Flusher::doFlush (this=0x3a427b0) at flusher.cc:240 #10 0x00007fbfed530169 in Flusher::step (this=0x3a427b0, d=..., tid=...) at flusher.cc:154 #11 0x00007fbfed530611 in FlusherStepper::callback (this=0x3a42780, d=..., t=...) at flusher.cc:23 #12 0x00007fbfed4de874 in Task::run (this=0x3a47de0, d=..., t=...) at dispatcher.hh:139 #13 0x00007fbfed4dd192 in Dispatcher::run (this=0x2b01f30) at dispatcher.cc:119 #14 0x00007fbfed4dd7f0 in launch_dispatcher_thread (arg=0x2b01f30) at dispatcher.cc:28 #15 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #16 0x00007fbff15516cd in clone () from /lib/libc.so.6 #17 0x0000000000000000 in ?? () Thread 10 (Thread 4659): #0 0x00007fbff17fb5e5 in __lll_unlock_wake () from /lib/libpthread.so.0 #1 0x00007fbff17f7af6 in _L_unlock_696 () from /lib/libpthread.so.0 #2 0x00007fbff17f7a33 in pthread_mutex_unlock () from /lib/libpthread.so.0 #3 0x00007fbfed4d4eed in Mutex::release (this=0x19a1c00) at mutex.hh:78 #4 0x00007fbfed4d501a in LockHolder::unlock (this=0x7fbfee057430) at locks.hh:58 #5 0x00007fbfed4d5031 in ~LockHolder (this=0x7fbfee057430, __in_chrg=) at locks.hh:41 #6 0x00007fbfed556ec7 in TapConnMap::disconnect (this=0x19a1c00, cookie=0x407a8e8, tapKeepAlive=300) at tapconnmap.cc:71 #7 0x00007fbfed52e031 in EventuallyPersistentEngine::handleDisconnect ( this=0x19a1660, cookie=0x407a8e8) at ep_engine.h:408 #8 0x00007fbfed50ec28 in EvpHandleDisconnect (cookie=0x407a8e8, type=ON_DISCONNECT, event_data=0x0, cb_data=0x19a1660) at ep_engine.cc:843 #9 0x00007fbff005fe48 in handle_disconnect (cookie=0x407a8e8, type=ON_DISCONNECT, event_data=0x0, cb_data=0x7fbff02661a0) at bucket_engine.c:859 #10 0x0000000000403847 in perform_callbacks (type=ON_DISCONNECT, data=0x0, c=0x407a8e8) at daemon/memcached.c:195 #11 0x00000000004068ad in conn_immediate_close (c=0x407a8e8) at daemon/memcached.c:5451 #12 0x0000000000407263 in event_handler (fd=, which=, arg=0x407a8e8) at daemon/memcached.c:5576 #13 0x00007fbff20ab3cd in event_process_active_single_queue (base=0x19b84a0, flags=) at event.c:1308 #14 event_process_active (base=0x19b84a0, flags=) at event.c:1375 #15 event_base_loop (base=0x19b84a0, flags=) at event.c:1572 #16 0x0000000000414824 in worker_libevent (arg=0x19b5fc0) at daemon/thread.c:304 #17 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #18 0x00007fbff15516cd in clone () from /lib/libc.so.6 #19 0x0000000000000000 in ?? () Thread 9 (Thread 4655): #0 0x00007fbff156112e in ?? () from /lib/libc.so.6 #1 0x00007fbff14ea3ac in ?? () from /lib/libc.so.6 #2 0x00007fbff14e8e48 in free () from /lib/libc.so.6 #3 0x00007fbfed4d57a0 in Blob::operator delete (p=0xa6c8d30) at item.hh:92 #4 0x00007fbfed4d7d7f in std::_Sp_counted_ptr::_M_dispose (this=0xa6c8d60) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:75 #5 0x00007fbfed4d478c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xa6c8d60) at /usr/include/c++/4.3/tr1_impl/boost_sp_counted_base.h:145 #6 0x00007fbfed4d47e2 in ~__shared_count (this=0xa6c8e80, __in_chrg=) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:320 #7 0x00007fbfed4d48cf in ~__shared_ptr (this=0xa6c8e78, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:206 #8 0x00007fbfed4d48e7 in ~shared_ptr (this=0xa6c8e78, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:822 #9 0x00007fbfed4d4903 in ~Item (this=0xa6c8e60, __in_chrg=) at item.hh:167 #10 0x00007fbfed4d7daf in ~QueuedItem (this=0xa6c8e50, __in_chrg=) at queueditem.hh:52 #11 0x00007fbfed4d7f1e in std::_Sp_counted_ptr::_M_dispose (this=0xa6c8eb0) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:75 #12 0x00007fbfed4d478c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xa6c8eb0) at /usr/include/c++/4.3/tr1_impl/boost_sp_counted_base.h:145 #13 0x00007fbfed4d47e2 in ~__shared_count (this=0xa6c8ee8, __in_chrg=) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:320 #14 0x00007fbfed4d47fd in ~__shared_ptr (this=0xa6c8ee0, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:206 #15 0x00007fbfed4d4815 in ~shared_ptr (this=0xa6c8ee0, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:822 #16 0x00007fbfed4d4831 in __gnu_cxx::new_allocator >::destroy (this=0x7fbff005ad07, __p=0xa6c8ee0) at /usr/include/c++/4.3/ext/new_allocator.h:118 #17 0x00007fbfed4d73cb in std::list, std::allocator > >::_M_erase (this=0x4213200, __position=...) at /usr/include/c++/4.3/bits/stl_list.h:1360 #18 0x00007fbfed4d7415 in std::list, std::allocator > >::erase (this=0x4213200, __position=...) at /usr/include/c++/4.3/bits/list.tcc:112 #19 0x00007fbfed4ceae0 in Checkpoint::queueDirty (this=0x42131e0, item=..., checkpointManager=0x61ed9a0) at checkpoint.cc:72 #20 0x00007fbfed4d0a13 in CheckpointManager::queueDirty (this=0x61ed9a0, item=..., vbucket=...) at checkpoint.cc:488 #21 0x00007fbfed4e2566 in EventuallyPersistentStore::queueDirty ( this=0x2aaf820, key=..., vbid=188, op=queue_op_set, value=..., flags=0, exptime=0, cas=1428513, rowid=1711) at ep.cc:1917 #22 0x00007fbfed4e35c6 in EventuallyPersistentStore::set (this=0x2aaf820, item=..., cookie=0x3bb9d48, force=true) at ep.cc:616 #23 0x00007fbfed50ab47 in EventuallyPersistentEngine::tapNotify ( this=0x19a1660, cookie=0x3bb9d48, engine_specific=0x16389b30, nengine=0, tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=215461, key=0x16389b38, nkey=45, flags=0, exptime=0, data=0x16389b65, ndata=20, vbucket=188) at ep_engine.cc:2098 #24 0x00007fbfed50b2fa in EvpTapNotify (handle=0x19a1660, cookie=0x3bb9d48, engine_specific=0x16389b30, nengine=0, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=215461, key=0x16389b38, nkey=45, flags=0, exptime=0, cas=1448834, data=0x16389b65, ndata=20, vbucket=188) at ep_engine.cc:805 #25 0x00007fbff0061757 in bucket_tap_notify (handle=0x7fbff02661a0, cookie=0x3bb9d48, engine_specific=0x16389b30, nengine=0, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=215461, key=0x16389b38, nkey=45, flags=0, exptime=0, cas=1448834, data=0x16389b65, ndata=20, vbucket=188) at bucket_engine.c:1527 #26 0x0000000000409a43 in process_bin_tap_packet (event=372880208, c=0x3bb9d48) at daemon/memcached.c:2703 #27 0x000000000040f03f in process_bin_packet (c=0x3bb9d48) at daemon/memcached.c:2790 #28 complete_nread_binary (c=0x3bb9d48) at daemon/memcached.c:3389 #29 0x000000000040f958 in complete_nread (c=0x3bb9d48) at daemon/memcached.c:3471 #30 conn_nread (c=0x3bb9d48) at daemon/memcached.c:5275 #31 0x0000000000407263 in event_handler (fd=, which=, arg=0x3bb9d48) at daemon/memcached.c:5576 #32 0x00007fbff20ab3cd in event_process_active_single_queue (base=0x19b6120, flags=) at event.c:1308 #33 event_process_active (base=0x19b6120, flags=) at event.c:1375 #34 event_base_loop (base=0x19b6120, flags=) at event.c:1572 #35 0x0000000000414824 in worker_libevent (arg=0x19b5bc0) at daemon/thread.c:304 #36 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #37 0x00007fbff15516cd in clone () from /lib/libc.so.6 #38 0x0000000000000000 in ?? () Thread 8 (Thread 4735): #0 0x00007fbff17f8bc9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007fbfed4dfb56 in SyncObject::wait (this=0x3a416d0, tv=...) at syncobject.hh:42 #2 0x00007fbfed4dbef9 in IdleTask::run (this=0x3a42700, d=...) at dispatcher.cc:244 #3 0x00007fbfed4dd192 in Dispatcher::run (this=0x3a416c0) at dispatcher.cc:119 #4 0x00007fbfed4dd7f0 in launch_dispatcher_thread (arg=0x3a416c0) at dispatcher.cc:28 #5 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #6 0x00007fbff15516cd in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 7 (Thread 4733): #0 0x00007fbff17f885c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00007fbfed4e0bea in SyncObject::wait (this=0x3a09e90) at syncobject.hh:31 #2 0x00007fbfed4dcec0 in Dispatcher::run (this=0x3a09e80) at dispatcher.cc:85 #3 0x00007fbfed4dd7f0 in launch_dispatcher_thread (arg=0x3a09e80) at dispatcher.cc:28 #4 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #5 0x00007fbff15516cd in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (Thread 4657): #0 0x00007fbff156112e in ?? () from /lib/libc.so.6 #1 0x00007fbff14ea3ac in ?? () from /lib/libc.so.6 #2 0x00007fbff14e8e48 in free () from /lib/libc.so.6 #3 0x00007fbfed4d57a0 in Blob::operator delete (p=0xce9d4c0) at item.hh:92 #4 0x00007fbfed4d7d7f in std::_Sp_counted_ptr::_M_dispose (this=0xce9d4f0) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:75 #5 0x00007fbfed4d478c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xce9d4f0) at /usr/include/c++/4.3/tr1_impl/boost_sp_counted_base.h:145 #6 0x00007fbfed4d47e2 in ~__shared_count (this=0xce9d610, __in_chrg=) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:320 #7 0x00007fbfed4d48cf in ~__shared_ptr (this=0xce9d608, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:206 #8 0x00007fbfed4d48e7 in ~shared_ptr (this=0xce9d608, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:822 #9 0x00007fbfed4d4903 in ~Item (this=0xce9d5f0, __in_chrg=) at item.hh:167 #10 0x00007fbfed4d7daf in ~QueuedItem (this=0xce9d5e0, __in_chrg=) at queueditem.hh:52 #11 0x00007fbfed4d7f1e in std::_Sp_counted_ptr::_M_dispose (this=0xce9d640) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:75 #12 0x00007fbfed4d478c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0xce9d640) at /usr/include/c++/4.3/tr1_impl/boost_sp_counted_base.h:145 #13 0x00007fbfed4d47e2 in ~__shared_count (this=0xce9d678, __in_chrg=) at /usr/include/c++/4.3/bits/boost_sp_shared_count.h:320 #14 0x00007fbfed4d47fd in ~__shared_ptr (this=0xce9d670, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:206 #15 0x00007fbfed4d4815 in ~shared_ptr (this=0xce9d670, __in_chrg=) at /usr/include/c++/4.3/tr1_impl/boost_shared_ptr.h:822 #16 0x00007fbfed4d4831 in __gnu_cxx::new_allocator >::destroy (this=0x7fbfef058d07, __p=0xce9d670) at /usr/include/c++/4.3/ext/new_allocator.h:118 #17 0x00007fbfed4d73cb in std::list, std::allocator > >::_M_erase (this=0x4790f00, __position=...) at /usr/include/c++/4.3/bits/stl_list.h:1360 #18 0x00007fbfed4d7415 in std::list, std::allocator > >::erase (this=0x4790f00, __position=...) at /usr/include/c++/4.3/bits/list.tcc:112 #19 0x00007fbfed4ceae0 in Checkpoint::queueDirty (this=0x4790ee0, item=..., checkpointManager=0x4790c00) at checkpoint.cc:72 #20 0x00007fbfed4d0a13 in CheckpointManager::queueDirty (this=0x4790c00, item=..., vbucket=...) at checkpoint.cc:488 #21 0x00007fbfed4e2566 in EventuallyPersistentStore::queueDirty ( this=0x2aaf820, key=..., vbid=18, op=queue_op_set, value=..., flags=0, exptime=0, cas=1428889, rowid=3277) at ep.cc:1917 #22 0x00007fbfed4e35c6 in EventuallyPersistentStore::set (this=0x2aaf820, item=..., cookie=0x4099dc8, force=true) at ep.cc:616 #23 0x00007fbfed50ab47 in EventuallyPersistentEngine::tapNotify ( this=0x19a1660, cookie=0x4099dc8, engine_specific=0x1628f6b8, nengine=0, tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=168383, key=0x1628f6c0, nkey=44, flags=0, exptime=0, data=0x1628f6ec, ndata=20, vbucket=18) at ep_engine.cc:2098 #24 0x00007fbfed50b2fa in EvpTapNotify (handle=0x19a1660, cookie=0x4099dc8, engine_specific=0x1628f6b8, nengine=0, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=168383, key=0x1628f6c0, nkey=44, flags=0, exptime=0, cas=2096041, data=0x1628f6ec, ndata=20, vbucket=18) at ep_engine.cc:805 #25 0x00007fbff0061757 in bucket_tap_notify (handle=0x7fbff02661a0, cookie=0x4099dc8, engine_specific=0x1628f6b8, nengine=0, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=168383, key=0x1628f6c0, nkey=44, flags=0, exptime=0, cas=2096041, data=0x1628f6ec, ndata=20, vbucket=18) at bucket_engine.c:1527 #26 0x0000000000409a43 in process_bin_tap_packet (event=3699311328, c=0x4099dc8) at daemon/memcached.c:2703 #27 0x000000000040f03f in process_bin_packet (c=0x4099dc8) at daemon/memcached.c:2790 #28 complete_nread_binary (c=0x4099dc8) at daemon/memcached.c:3389 #29 0x000000000040f958 in complete_nread (c=0x4099dc8) at daemon/memcached.c:3471 #30 conn_nread (c=0x4099dc8) at daemon/memcached.c:5275 #31 0x0000000000407263 in event_handler (fd=, which=, arg=0x4099dc8) at daemon/memcached.c:5576 #32 0x00007fbff20ab3cd in event_process_active_single_queue (base=0x19b7260, flags=) at event.c:1308 #33 event_process_active (base=0x19b7260, flags=) at event.c:1375 #34 event_base_loop (base=0x19b7260, flags=) at event.c:1572 #35 0x0000000000414824 in worker_libevent (arg=0x19b5dc0) at daemon/thread.c:304 #36 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #37 0x00007fbff15516cd in clone () from /lib/libc.so.6 #38 0x0000000000000000 in ?? () Thread 5 (Thread 4656): #0 0x00007fbff156112e in ?? () from /lib/libc.so.6 #1 0x00007fbff14ea3ac in ?? () from /lib/libc.so.6 #2 0x00007fbff14e8e48 in free () from /lib/libc.so.6 #3 0x00007fbfed4d5379 in __gnu_cxx::new_allocator > >::deallocate (this=0x43f2b60, __p=0x8d852b0) at /usr/include/c++/4.3/ext/new_allocator.h:98 #4 0x00007fbfed4d53a1 in std::_List_base, std::allocator > >::_M_put_node (this=0x43f2b60, __p=0x8d852b0) at /usr/include/c++/4.3/bits/stl_list.h:318 #5 0x00007fbfed4d73e1 in std::list, std::allocator > >::_M_erase (this=0x43f2b60, __position=...) at /usr/include/c++/4.3/bits/stl_list.h:1361 #6 0x00007fbfed4d7415 in std::list, std::allocator > >::erase (this=0x43f2b60, __position=...) at /usr/include/c++/4.3/bits/list.tcc:112 #7 0x00007fbfed4ceae0 in Checkpoint::queueDirty (this=0x43f2b40, item=..., checkpointManager=0x4673ca0) at checkpoint.cc:72 #8 0x00007fbfed4d0a13 in CheckpointManager::queueDirty (this=0x4673ca0, item=..., vbucket=...) at checkpoint.cc:488 #9 0x00007fbfed4e2566 in EventuallyPersistentStore::queueDirty ( this=0x2aaf820, key=..., vbid=50, op=queue_op_set, value=..., flags=0, exptime=0, cas=1428890, rowid=57) at ep.cc:1917 #10 0x00007fbfed4e35c6 in EventuallyPersistentStore::set (this=0x2aaf820, item=..., cookie=0x4127248, force=true) at ep.cc:616 #11 0x00007fbfed50ab47 in EventuallyPersistentEngine::tapNotify ( this=0x19a1660, cookie=0x4127248, engine_specific=0x16370880, nengine=0, tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=160005, key=0x16370888, nkey=45, flags=0, exptime=0, data=0x163708b5, ndata=20, vbucket=50) at ep_engine.cc:2098 #12 0x00007fbfed50b2fa in EvpTapNotify (handle=0x19a1660, cookie=0x4127248, engine_specific=0x16370880, nengine=0, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=160005, key=0x16370888, nkey=45, flags=0, exptime=0, cas=3047246, data=0x163708b5, ndata=20, vbucket=50) at ep_engine.cc:805 #13 0x00007fbff0061757 in bucket_tap_notify (handle=0x7fbff02661a0, cookie=0x4127248, engine_specific=0x16370880, nengine=0, ttl=254 '\376', tap_flags=0, tap_event=TAP_MUTATION, tap_seqno=160005, key=0x16370888, nkey=45, flags=0, exptime=0, cas=3047246, data=0x163708b5, ndata=20, vbucket=50) at bucket_engine.c:1527 #14 0x0000000000409a43 in process_bin_tap_packet (event=372741760, c=0x4127248) at daemon/memcached.c:2703 #15 0x000000000040f03f in process_bin_packet (c=0x4127248) at daemon/memcached.c:2790 #16 complete_nread_binary (c=0x4127248) at daemon/memcached.c:3389 #17 0x000000000040f958 in complete_nread (c=0x4127248) at daemon/memcached.c:3471 #18 conn_nread (c=0x4127248) at daemon/memcached.c:5275 #19 0x0000000000407263 in event_handler (fd=, which=, arg=0x4127248) at daemon/memcached.c:5576 #20 0x00007fbff20ab3cd in event_process_active_single_queue (base=0x19b69c0, flags=) at event.c:1308 #21 event_process_active (base=0x19b69c0, flags=) at event.c:1375 #22 event_base_loop (base=0x19b69c0, flags=) at event.c:1572 #23 0x0000000000414824 in worker_libevent (arg=0x19b5cc0) at daemon/thread.c:304 #24 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #25 0x00007fbff15516cd in clone () from /lib/libc.so.6 #26 0x0000000000000000 in ?? () Thread 4 (Thread 4654): #0 0x00007fbff151535d in nanosleep () from /lib/libc.so.6 #1 0x00007fbff15151d0 in sleep () from /lib/libc.so.6 #2 0x0000000000416618 in check_isasl_db_thread (arg=) at daemon/isasl.c:220 #3 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #4 0x00007fbff15516cd in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 3 (Thread 4653): #0 0x00007fbff154347d in read () from /lib/libc.so.6 #1 0x00007fbff14de348 in _IO_file_underflow () from /lib/libc.so.6 #2 0x00007fbff14dfeee in _IO_default_uflow () from /lib/libc.so.6 #3 0x00007fbff14d6c7b in getc () from /lib/libc.so.6 #4 0x00007fbff1269965 in check_stdin_thread (arg=0x403b70) at extensions/daemon/stdin_check.c:19 #5 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #6 0x00007fbff15516cd in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 2 (Thread 4652): #0 0x00007fbff1551cc3 in epoll_wait () from /lib/libc.so.6 #1 0x00007fbff20bc95c in epoll_dispatch (base=0x19aab20, tv=) at epoll.c:404 #2 0x00007fbff20ab2cc in event_base_loop (base=0x19aab20, flags=) at event.c:1558 #3 0x000000000040ae28 in main (argc=, argv=0x7fffb88b0f78) at daemon/memcached.c:7459 Thread 1 (Thread 4736): #0 0x00007fbff149ea75 in raise () from /lib/libc.so.6 #1 0x00007fbff14a25c0 in abort () from /lib/libc.so.6 #2 0x00007fbff1497941 in __assert_fail () from /lib/libc.so.6 #3 0x00007fbff0062cd4 in bucket_engine_release_cookie (cookie=0x407a8e8) at bucket_engine.c:1977 #4 0x00007fbfed5583f3 in TapConnMap::notifyIOThreadMain (this=0x19a1c00) at tapconnmap.cc:441 #5 0x00007fbfed502ff2 in EventuallyPersistentEngine::notifyTapIoThread ( this=0x19a1660) at ep_engine.cc:3568 #6 0x00007fbfed508a67 in EvpNotifyTapIo (arg=0x19a1660) at ep_engine.cc:882 #7 0x00007fbff17f39ca in start_thread () from /lib/libpthread.so.0 #8 0x00007fbff15516cd in clone () from /lib/libc.so.6 #9 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 membase membase 31 2011-05-06 20:24 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 685d6e2b61ffe61cad24b15050d44c05 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 membase membase 21 2011-05-06 20:24 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 ae6ec7b927064890ae7d820f067d8fca /opt/membase/lib/libevent-2.0.so.5 /lib/libdl.so.2: lrwxrwxrwx 1 root root 15 2010-05-28 15:10 /lib/libdl.so.2 -> libdl-2.11.1.so 4fc2038540e79999571aa4a338ac8692 /lib/libdl.so.2 /lib/libm.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:10 /lib/libm.so.6 -> libm-2.11.1.so 54f6f084ce7ce998d5ae8efaf5daea70 /lib/libm.so.6 /lib/librt.so.1: lrwxrwxrwx 1 root root 15 2010-05-28 15:10 /lib/librt.so.1 -> librt-2.11.1.so 1cfe1e6a4c1630a2e050303652c3341d /lib/librt.so.1 /lib/libpthread.so.0: lrwxrwxrwx 1 root root 20 2010-05-28 15:10 /lib/libpthread.so.0 -> libpthread-2.11.1.so deb7c9592f790b5ec2910b627df2d480 /lib/libpthread.so.0 /lib/libc.so.6: lrwxrwxrwx 1 root root 14 2010-05-28 15:10 /lib/libc.so.6 -> libc-2.11.1.so 3a9c344619f5243066e844d540de6867 /lib/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 12 2010-05-28 15:10 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so 25dd1c7aa9cbf6535dc0ec73f6b94b62 /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rw-r--r-- 1 membase membase 17427 2011-05-06 19:48 /opt/membase/lib/memcached/stdin_term_handler.so 02e2ef2d3374e35e45965234c83f552c /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 membase membase 22 2011-05-06 20:24 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 85553d81662039ffa267df512aa92a6c /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 membase membase 11 2011-05-06 20:24 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 1.6.5.3_237_g6656d4b EP engine v1.6.5.3_237_g6656d4b f705c55f1f5d9acfa7bbef6506850654 /opt/membase/lib/memcached/ep.so /usr/lib/libstdc++.so.6: lrwxrwxrwx 1 root root 19 2010-05-28 12:00 /usr/lib/libstdc++.so.6 -> libstdc++.so.6.0.13 145a441e95ff795ace078ba78840b91f /usr/lib/libstdc++.so.6 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 92552 2010-03-26 17:12 /lib/libgcc_s.so.1 2fb8dc6079174e77e399d0d5b57b11e5 /lib/libgcc_s.so.1