Basic crash dump analysis of /tmp/backup/core.memcached.19349. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 392488 Jun 3 03:48 /opt/membase/bin/memcached -rw------- 1 membase membase 396550144 Jun 3 06:36 /tmp/backup/core.memcached.19349 b913e0e184128d9354565e7ae7ddf56e /opt/membase/bin/memcached aa7c85fbd23ff6416729c708ec9c6930 /tmp/backup/core.memcached.19349 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Amazon Linux (7.1-29.9.amzn1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-amazon-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 19363] [New Thread 19367] [New Thread 19364] [New Thread 19369] [New Thread 19358] [New Thread 19370] [New Thread 19359] [New Thread 19349] [New Thread 19360] [New Thread 19362] [New Thread 19361] [New Thread 19368] warning: .dynamic section for "/opt/membase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?) Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x00007f7d31e386b5 in raise () from /lib64/libc.so.6 Thread 12 (Thread 19368): #0 0x00007f7d31edfd6d in fdatasync () from /lib64/libc.so.6 #1 0x00007f7d2df070d9 in unixCurrentTimeInt64 (NotUsed=0x33, piNow=0x7f7d280517e8) at embedded/sqlite3.c:27593 #2 0x00007f7d280517e8 in ?? () #3 0x00000000000076b8 in ?? () #4 0x00007f7d2df22108 in walIteratorInit (pWal=0x7f7d280517e8, sync_flags=2, nBuf=1024, zBuf=0x7f7d2804deb8 "\r") at embedded/sqlite3.c:42305 #5 walCheckpoint (pWal=0x7f7d280517e8, sync_flags=2, nBuf=1024, zBuf=0x7f7d2804deb8 "\r") at embedded/sqlite3.c:42367 #6 sqlite3WalCheckpoint (pWal=0x7f7d280517e8, sync_flags=2, nBuf=1024, zBuf=0x7f7d2804deb8 "\r") at embedded/sqlite3.c:43431 #7 0x00007f7d2df512f7 in findBtree (pErrorDb=0x7f7d280403e8, pDb=0x7f7d28040448, zDb=) at embedded/sqlite3.c:52722 #8 0x00007f7d2df514b1 in sqlite3_backup_init (pDestDb=0x7f7d2804d570, zDestDb=0x7f7d2803fa68 "@D\037.}\177", pSrcDb=0x65, zSrcDb=0x3e8
) at embedded/sqlite3.c:52790 #9 0x00007f7d2df8bacf in sqlite3Prepare (db=0x7f7d2df8cdc7, zSql=0x0, nBytes=0, saveSqlFlag=0, pReprepare=, ppStmt=0x0, pzTail=0x7f7d00000000) at embedded/sqlite3.c:86452 #10 0x00007f7d2defbb8e in PreparedStatement::bind (this=, pos=, s=) at sqlite-pst.cc:43 #11 0x0000000000000009 in ?? () #12 0x00007f7d1887a7e0 in ?? () #13 0x00007f7d1887a7d0 in ?? () #14 0x0000000000000009 in ?? () #15 0x00007f7d2def91a1 in ~basic_ifstream (this=0x7f7d32398e7f, filename=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/fstream:240 #16 SqliteEvaluator::eval (this=0x7f7d32398e7f, filename=) at sqlite-eval.cc:55 #17 0x0000000000000000 in ?? () Thread 11 (Thread 19361): #0 0x00007f7d31ee74a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7d32a46c28 in epoll_dispatch (base=0x23189c0, tv=) at epoll.c:404 #2 0x00007f7d32a35a4c in event_base_loop (base=0x23189c0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x2317cc0) at daemon/thread.c:304 #4 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 19362): #0 0x00007f7d31ee74a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7d32a46c28 in epoll_dispatch (base=0x2319260, tv=) at epoll.c:404 #2 0x00007f7d32a35a4c in event_base_loop (base=0x2319260, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x2317dc0) at daemon/thread.c:304 #4 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 19360): #0 0x00007f7d31ee74a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7d32a46c28 in epoll_dispatch (base=0x2318120, tv=) at epoll.c:404 #2 0x00007f7d32a35a4c in event_base_loop (base=0x2318120, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x2317bc0) at daemon/thread.c:304 #4 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 19349): #0 0x00007f7d31ee74a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7d32a46c28 in epoll_dispatch (base=0x230cb10, tv=) at epoll.c:404 #2 0x00007f7d32a35a4c in event_base_loop (base=0x230cb10, flags=0) at event.c:1558 #3 0x0000000000409974 in main (argc=, argv=) at daemon/memcached.c:7459 Thread 7 (Thread 19359): #0 0x00007f7d31eac46d in nanosleep () from /lib64/libc.so.6 #1 0x00007f7d31eac2e0 in sleep () from /lib64/libc.so.6 #2 0x0000000000414f58 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 19370): #0 0x00007f7d321837a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7d2dea6f37 in release (this=0x7f7d28036d60, when=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/boost_shared_ptr.h:153 #2 ~shared_count (this=0x7f7d28036d60, when=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/boost_shared_ptr.h:277 #3 ~shared_ptr (this=0x7f7d28036d60, when=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/boost_shared_ptr.h:486 #4 EventuallyPersistentEngine::flush (this=0x7f7d28036d60, when=) at ep_engine.cc:1847 #5 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 19358): #0 0x00007f7d31ed9b5d in read () from /lib64/libc.so.6 #1 0x00007f7d31e77228 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00007f7d31e78d2e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00007f7d31e7424b in getc () from /lib64/libc.so.6 #4 0x00007f7d31c05855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19 #5 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 4 (Thread 19369): #0 0x00007f7d321837a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7d2de856b0 in Dispatcher::start (this=0x7f7d2a017804) at dispatcher.cc:39 #2 0x000000004de880f5 in ?? () #3 0x000000000e4c4f10 in ?? () #4 0x00007f7d2c91ddf0 in ?? () #5 0x00007f7d2de81e27 in ~pair (this=0x7f7d2c91e700, k=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_pair.h:69 #6 Internal::map_base, std::allocator >, std::pair, std::allocator > const, index_entry>, Internal::extract1st, std::allocator > const, index_entry> >, true, std::tr1::hashtable, std::allocator >, std::pair, std::allocator > const, index_entry>, std::allocator, std::allocator > const, index_entry> >, Internal::extract1st, std::allocator > const, index_entry> >, std::equal_to, std::allocator > >, std::tr1::hash, std::allocator > >, Internal::mod_range_hashing, Internal::default_ranged_hash, Internal::prime_rehash_policy, false, false, true> >::operator[] (this=0x7f7d2c91e700, k=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/hashtable:691 #7 0x0000000000000000 in ?? () Thread 3 (Thread 19364): #0 0x00007f7d31ee74a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f7d32a46c28 in epoll_dispatch (base=0x231a4a0, tv=) at epoll.c:404 #2 0x00007f7d32a35a4c in event_base_loop (base=0x231a4a0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x2317fc0) at daemon/thread.c:304 #4 0x00007f7d3217f7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f7d31ee6ead in clone () from /lib64/libc.so.6 Thread 2 (Thread 19367): #0 0x00007f7d3218343c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f7d2de87823 in Dispatcher::completeNonDaemonTasks ( this=0x7f7d2d920700) at dispatcher.cc:234 #2 0x0000000000000000 in ?? () Thread 1 (Thread 19363): #0 0x00007f7d31e386b5 in raise () from /lib64/libc.so.6 #1 0x00007f7d31e39e95 in abort () from /lib64/libc.so.6 #2 0x00007f7d31e31645 in __assert_fail () from /lib64/libc.so.6 #3 0x00007f7d2de7eb06 in get (this=0x7f7d2f1f4da0, forceCreation=false, timeBound=224) at atomic.hh:109 #4 operator unsigned int (this=0x7f7d2f1f4da0, forceCreation=false, timeBound=224) at atomic.hh:118 #5 CheckpointManager::checkOpenCheckpoint_UNLOCKED (this=0x7f7d2f1f4da0, forceCreation=false, timeBound=224) at checkpoint.cc:710 #6 0x00007f7d20f583f0 in ?? () #7 0x000000000000029f in ?? () #8 0x00007f7d20f583e0 in ?? () #9 0x00007f7d2044b710 in ?? () #10 0x00007f7d28307500 in ?? () #11 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 Jun 3 08:07 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 c7fc13e0591be3155fea1efcd88c0430 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 Jun 3 08:07 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 62b1f1fede447a7642ca0f09d84eebe6 /opt/membase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 13 Feb 24 16:30 /lib64/libdl.so.2 -> libdl-2.12.so 643aa33f4e56d9dc502a3915703335a0 /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 12 Feb 24 16:30 /lib64/libm.so.6 -> libm-2.12.so b496148ca454ca96f9e39c88730b2f7a /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 13 Feb 24 16:30 /lib64/librt.so.1 -> librt-2.12.so d399fe4ea992cffb7227bbbc5dd827e6 /lib64/librt.so.1 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 18 Feb 24 16:30 /lib64/libpthread.so.0 -> libpthread-2.12.so e3cc21f2a81e11035cfadb97580867ef /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 12 Feb 24 16:30 /lib64/libc.so.6 -> libc-2.12.so b3e39b32eddb15662ba7d1c700a65bf0 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 10 Feb 24 16:30 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so bdfe53856f4614f5fb1a4fcad54d235c /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15322 Jun 3 03:48 /opt/membase/lib/memcached/stdin_term_handler.so 2b46627c47610208b90a34190ab97bb8 /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 Jun 3 08:07 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 bcffcfd18dcc1f8660bc8347f3eae2b3 /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 Jun 3 08:07 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.6.5.3_350_g2bdfd9e 1.6.5.3_350_g2bdfd9e bee1ceb429dd303b34c50756e12ba46f /opt/membase/lib/memcached/ep.so /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 19 Feb 24 16:30 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 0eb10e4f59017e1796fc0b1e227b9467 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Feb 24 16:30 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1 704445fcf4f7f80bf04c446957c31862 /lib64/libgcc_s.so.1