Basic crash dump analysis of /tmp//core.memcached.11894. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 392488 May 25 10:27 /opt/membase/./bin/memcached -rw------- 1 membase membase 676552704 May 25 16:07 /tmp//core.memcached.11894 b913e0e184128d9354565e7ae7ddf56e /opt/membase/./bin/memcached abb42dbd6e7fb61a0c0fcf0fbef1d0b2 /tmp//core.memcached.11894 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 11917] [New Thread 11916] [New Thread 11915] [New Thread 11914] [New Thread 11905] [New Thread 11904] [New Thread 11903] [New Thread 11898] [New Thread 11897] [New Thread 11896] [New Thread 11895] [New Thread 11894] Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x00000037fba30265 in raise () from /lib64/libc.so.6 Thread 13 (Thread 11894): #0 0x00000037fbad4358 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b60b4675c28 in epoll_dispatch (base=0x1b14da80, tv=) at epoll.c:404 #2 0x00002b60b4664a4c in event_base_loop (base=0x1b14da80, flags=0) at event.c:1558 #3 0x0000000000409974 in main (argc=, argv=) at daemon/memcached.c:7459 Thread 12 (Thread 11895): #0 0x00000037fbac627b in read () from /lib64/libc.so.6 #1 0x00000037fba6cc07 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00000037fba6d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00000037fba68e8b in getc () from /lib64/libc.so.6 #4 0x00002b60b48a9855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19 #5 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 11 (Thread 11896): #0 0x00000037fba9a3f1 in nanosleep () from /lib64/libc.so.6 #1 0x00000037fba9a214 in sleep () from /lib64/libc.so.6 #2 0x0000000000414f58 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #4 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 10 (Thread 11897): #0 0x00000037fbad4358 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b60b4675c28 in epoll_dispatch (base=0x1b159090, tv=) at epoll.c:404 #2 0x00002b60b4664a4c in event_base_loop (base=0x1b159090, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x1b158b30) at daemon/thread.c:304 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 9 (Thread 11898): #0 0x00000037fbad4358 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b60b4675c28 in epoll_dispatch (base=0x1b159930, tv=) at epoll.c:404 #2 0x00002b60b4664a4c in event_base_loop (base=0x1b159930, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x1b158c30) at daemon/thread.c:304 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 8 (Thread 11903): #0 0x00000037fbad4358 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b60b4675c28 in epoll_dispatch (base=0x1b15a1d0, tv=) at epoll.c:404 #2 0x00002b60b4664a4c in event_base_loop (base=0x1b15a1d0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x1b158d30) at daemon/thread.c:304 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 7 (Thread 11904): #0 0x00000037fbad4358 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b60b4675c28 in epoll_dispatch (base=0x1b15aa70, tv=) at epoll.c:404 #2 0x00002b60b4664a4c in event_base_loop (base=0x1b15aa70, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x1b158e30) at daemon/thread.c:304 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 6 (Thread 11905): #0 0x00000037fbad4358 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b60b4675c28 in epoll_dispatch (base=0x1b15b410, tv=) at epoll.c:404 #2 0x00002b60b4664a4c in event_base_loop (base=0x1b15b410, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x1b158f30) at daemon/thread.c:304 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 5 (Thread 11914): #0 0x00000037fc60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaacfdcd3 in wait (this=0x1c41fbb0) at syncobject.hh:31 #2 Dispatcher::run (this=0x1c41fbb0) at dispatcher.cc:85 #3 0x00002aaaaacfe8e3 in launch_dispatcher_thread (arg=0x1c41fc04) at dispatcher.cc:28 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 4 (Thread 11915): #0 0x00000037fbacd587 in fdatasync () from /lib64/libc.so.6 #1 0x00002aaaaad7b4a9 in full_fsync (id=, flags=2) at embedded/sqlite3.c:25510 #2 unixSync (id=, flags=2) at embedded/sqlite3.c:25558 #3 0x00002aaaaad98358 in sqlite3WalFrames (pWal=0x1d9948f8, szPage=1024, pList=0x1d513cf8, nTruncate=18551, isCommit=1, sync_flags=2) at embedded/sqlite3.c:13413 #4 0x00002aaaaad99128 in sqlite3PagerCommitPhaseOne (pPager=0x1d994498, zMaster=0x0, noSync=0) at embedded/sqlite3.c:37082 #5 0x00002aaaaad9f627 in sqlite3BtreeCommitPhaseOne (p=0x1d80d7a8, zMaster=0x0) at embedded/sqlite3.c:47605 #6 0x00002aaaaadd0d4c in vdbeCommit (db=0x1d5e1668, p=) at embedded/sqlite3.c:56090 #7 0x00002aaaaadd27bd in sqlite3VdbeHalt (p=0x1ee32d68) at embedded/sqlite3.c:56514 #8 0x00002aaaaae2133a in sqlite3VdbeExec (p=0x1ee32d68) at embedded/sqlite3.c:60326 #9 0x00002aaaaae00d5a in sqlite3Step (pStmt=0x1ee32d68) at embedded/sqlite3.c:57947 #10 sqlite3_step (pStmt=0x1ee32d68) at embedded/sqlite3.c:58011 #11 0x00002aaaaad6f999 in PreparedStatement::execute (this=0x45b18690) at sqlite-pst.cc:73 #12 0x00002aaaaad70e48 in SqliteStrategy::execute ( this=, query=) at sqlite-strategies.cc:151 #13 0x00002aaaaad712b4 in ShardedByVBucketSqliteStrategy::destroyTables ( this=0x1b212c00) at sqlite-strategies.cc:417 #14 0x00002aaaaad6da4f in StrategicSqlite3::reset (this=0x1b212ce0) at sqlite-kvstore.cc:145 #15 0x00002aaaaad00fce in EventuallyPersistentStore::flushOneDeleteAll ( this=0x1b4e0730) at ep.cc:1733 #16 0x00002aaaaad0ac9d in EventuallyPersistentStore::flushOne ( this=0x1b4e0730, q=0x1b4e0860, rejectQueue=0x1d98f6f0) at ep.cc:1872 #17 0x00002aaaaad0ae8e in EventuallyPersistentStore::flushSome ( this=0x1b4e0730, q=0x1b4e0860, rejectQueue=0x1d98f6f0) at ep.cc:1489 #18 0x00002aaaaad40eea in Flusher::doFlush (this=0x1d1df2c0) at flusher.cc:242 #19 0x00002aaaaad4194a in Flusher::step (this=0x67, d=..., tid=Traceback (most recent call last): File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string return '%s (count %d) %s' % (self.typename, TypeError: int argument required ) at flusher.cc:156 #20 0x00002aaaaad421be in FlusherStepper::callback (this=0x1c41fe40, d=..., t=) at flusher.cc:23 #21 0x00002aaaaacff2df in Task::run (this=, d=..., t=) at dispatcher.hh:139 #22 0x00002aaaaacfdeaf in Dispatcher::run (this=0x1c2a20f0) at dispatcher.cc:119 #23 0x00002aaaaacfe8e3 in launch_dispatcher_thread (arg=0x67) at dispatcher.cc:28 #24 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #25 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 3 (Thread 11916): #0 0x00000037fc60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaacfbb60 in wait (this=0x1d1df200, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x1d1df200, d=...) at dispatcher.cc:244 #3 0x00002aaaaacfdeaf in Dispatcher::run (this=0x1d1ddff0) at dispatcher.cc:119 #4 0x00002aaaaacfe8e3 in launch_dispatcher_thread (arg=0x1d1de044) at dispatcher.cc:28 #5 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 11917): #0 0x00000037fc60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad1ce07 in wait (this=0x1b209e10) at syncobject.hh:42 #2 wait (this=0x1b209e10) at tapconnmap.hh:185 #3 EventuallyPersistentEngine::notifyTapIoThread (this=0x1b209e10) at ep_engine.cc:3695 #4 0x00002aaaaad1cf71 in EvpNotifyTapIo (arg=0x1b209e10) at ep_engine.cc:894 #5 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 Thread 1 (Thread 11959): #0 0x00000037fba30265 in raise () from /lib64/libc.so.6 #1 0x00000037fba31d10 in abort () from /lib64/libc.so.6 #2 0x00000037fba296e6 in __assert_fail () from /lib64/libc.so.6 #3 0x00002aaaaaaaebc1 in engine_shutdown_thread (arg=0x1b1d5bd0) at bucket_engine.c:1401 #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 May 25 15:39 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 c7fc13e0591be3155fea1efcd88c0430 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 May 25 15:39 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 62b1f1fede447a7642ca0f09d84eebe6 /opt/membase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 12 Jan 25 14:32 /lib64/libdl.so.2 -> libdl-2.5.so b4ab0a602eb5dcdfeb5ed6b324ee06be /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 11 Jan 25 14:32 /lib64/libm.so.6 -> libm-2.5.so ea82374899f156976079dcd903604fd7 /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 12 Jan 25 14:32 /lib64/librt.so.1 -> librt-2.5.so 30cbb89ba718e8fed2dd9e8b30a8a76d /lib64/librt.so.1 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 17 Jan 25 14:32 /lib64/libpthread.so.0 -> libpthread-2.5.so da167e96aec879c4f69dcd6f7e57dbc3 /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 11 Jan 25 14:32 /lib64/libc.so.6 -> libc-2.5.so 9cc61e6758baa46fcaddc746bc34185e /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 9 Jan 25 14:32 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so 58c68da42c5d61a8b78fa1756e6cceeb /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15322 May 25 10:27 /opt/membase/lib/memcached/stdin_term_handler.so 2b46627c47610208b90a34190ab97bb8 /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 May 25 15:39 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 377424b09d1f3699b76e4143c862b296 /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 May 25 15:39 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.6.5.3_296_gbefddad 1.6.5.3_296_gbefddad 2ab497abd84f59398da82397f92c9e14 /opt/membase/lib/memcached/ep.so /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 18 Jan 25 16:13 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.8 6c38ca845aa5aed6dded78e1dd6ac378 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Jan 25 14:32 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1 a4ac4784c636e04de489f0369f271aa0 /lib64/libgcc_s.so.1