Basic crash dump analysis of /tmp/backup/core.memcached.19339. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 392488 Jun 3 03:48 /opt/membase/bin/memcached -rw------- 1 membase membase 453406720 Jun 3 06:36 /tmp/backup/core.memcached.19339 b913e0e184128d9354565e7ae7ddf56e /opt/membase/bin/memcached 8e12c5072b5aecc046678958326b26ed /tmp/backup/core.memcached.19339 memcached 1.4.4_461_gf99c147 memcached 1.4.4_461_gf99c147 VERSION 1.4.4_461_gf99c147 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Amazon Linux (7.1-29.9.amzn1) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-amazon-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/memcached...done. [New Thread 19351] [New Thread 19354] [New Thread 19359] [New Thread 19362] [New Thread 19339] [New Thread 19348] [New Thread 19349] [New Thread 19361] [New Thread 19353] [New Thread 19350] [New Thread 19352] [New Thread 19360] warning: .dynamic section for "/opt/membase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?) Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/membase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so Reading symbols from /opt/membase/lib/memcached/ep.so...done. Loaded symbols for /opt/membase/lib/memcached/ep.so Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'. Program terminated with signal 6, Aborted. #0 0x00007f6d1c1686b5 in raise () from /lib64/libc.so.6 Thread 12 (Thread 19360): #0 0x00007f6d1c4b6bcd in fsync () from /lib64/libpthread.so.0 #1 0x00007f6d182617b3 in checkTreePage (pCheck=0x0, iPage=5, zParentContext=0x0, pnParentMinKey=0x7f6d032e0428, pnParentMaxKey=0x0) at embedded/sqlite3.c:52184 #2 0x00007f6cfc879598 in ?? () #3 0x00007f6cfd1f5198 in ?? () #4 0x0000000000000000 in ?? () Thread 11 (Thread 19352): #0 0x00007f6d1c2174a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d1cd76c28 in epoll_dispatch (base=0x20e7260, tv=) at epoll.c:404 #2 0x00007f6d1cd65a4c in event_base_loop (base=0x20e7260, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x20e5dc0) at daemon/thread.c:304 #4 0x00007f6d1c4af7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d1c216ead in clone () from /lib64/libc.so.6 Thread 10 (Thread 19350): #0 0x00007f6d1c2174a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d1cd76c28 in epoll_dispatch (base=0x20e6120, tv=) at epoll.c:404 #2 0x00007f6d1cd65a4c in event_base_loop (base=0x20e6120, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x20e5bc0) at daemon/thread.c:304 #4 0x00007f6d1c4af7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d1c216ead in clone () from /lib64/libc.so.6 Thread 9 (Thread 19353): #0 0x00007f6d1c2174a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d1cd76c28 in epoll_dispatch (base=0x20e7b00, tv=) at epoll.c:404 #2 0x00007f6d1cd65a4c in event_base_loop (base=0x20e7b00, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x20e5ec0) at daemon/thread.c:304 #4 0x00007f6d1c4af7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d1c216ead in clone () from /lib64/libc.so.6 Thread 8 (Thread 19361): #0 0x00007f6d1c4b37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d181b56b0 in Dispatcher::start (this=0x7f6d16017804) at dispatcher.cc:39 #2 0x000000004de880f5 in ?? () #3 0x0000000002bdc070 in ?? () #4 0x00007f6d02adfdf0 in ?? () #5 0x00007f6d181b1e27 in ~pair (this=0x7f6d02ae0700, k=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_pair.h:69 #6 Internal::map_base, std::allocator >, std::pair, std::allocator > const, index_entry>, Internal::extract1st, std::allocator > const, index_entry> >, true, std::tr1::hashtable, std::allocator >, std::pair, std::allocator > const, index_entry>, std::allocator, std::allocator > const, index_entry> >, Internal::extract1st, std::allocator > const, index_entry> >, std::equal_to, std::allocator > >, std::tr1::hash, std::allocator > >, Internal::mod_range_hashing, Internal::default_ranged_hash, Internal::prime_rehash_policy, false, false, true> >::operator[] (this=0x7f6d02ae0700, k=) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/hashtable:691 #7 0x0000000000000000 in ?? () Thread 7 (Thread 19349): #0 0x00007f6d1c1dc46d in nanosleep () from /lib64/libc.so.6 #1 0x00007f6d1c1dc2e0 in sleep () from /lib64/libc.so.6 #2 0x0000000000414f58 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00007f6d1c4af7e1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f6d1c216ead in clone () from /lib64/libc.so.6 Thread 6 (Thread 19348): #0 0x00007f6d1c209b5d in read () from /lib64/libc.so.6 #1 0x00007f6d1c1a7228 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00007f6d1c1a8d2e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00007f6d1c1a424b in getc () from /lib64/libc.so.6 #4 0x00007f6d1bf35855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19 #5 0x00007f6d1c4af7e1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f6d1c216ead in clone () from /lib64/libc.so.6 Thread 5 (Thread 19339): #0 0x00007f6d1c2174a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d1cd76c28 in epoll_dispatch (base=0x20dab10, tv=) at epoll.c:404 #2 0x00007f6d1cd65a4c in event_base_loop (base=0x20dab10, flags=0) at event.c:1558 #3 0x0000000000409974 in main (argc=, argv=) at daemon/memcached.c:7459 Thread 4 (Thread 19362): #0 0x00007f6d1c4b37a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d1822334e in ~RCPtr (this=, tp=0x18) at atomic.hh:340 #2 TapConnMap::removeTapCursors_UNLOCKED (this=, tp=0x18) at tapconnmap.cc:164 #3 0x000000000bfe4830 in ?? () #4 0x00007f6d03fe62b8 in ?? () from /usr/lib64/libstdc++.so.6 #5 0x00007f6d03fe66d0 in vtable for std::basic_streambuf > () from /usr/lib64/libstdc++.so.6 #6 0x00007f6cf8000c18 in ?? () #7 0x00007f6cf8000c18 in ?? () #8 0x00007f6cf8000c19 in ?? () #9 0x00007f6cf8000c18 in ?? () #10 0x00007f6cf8000c40 in ?? () #11 0x00007f6cf8000e18 in ?? () #12 0x00007f6d03fed180 in ?? () from /usr/lib64/libstdc++.so.6 #13 0x0000000000000018 in ?? () #14 0x00007f6cf8000c18 in ?? () #15 0x00007f6d03fe3370 in vtable for std::ios_base () from /usr/lib64/libstdc++.so.6 #16 0x0000000000000006 in ?? () #17 0x0000000000000000 in ?? () Thread 3 (Thread 19359): #0 0x00007f6d1c4b343c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6d181b7823 in Dispatcher::completeNonDaemonTasks (this=0x7f6d03ae2700) at dispatcher.cc:234 #2 0x0000000000000000 in ?? () Thread 2 (Thread 19354): #0 0x00007f6d1c2174a3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6d1cd76c28 in epoll_dispatch (base=0x20e84a0, tv=) at epoll.c:404 #2 0x00007f6d1cd65a4c in event_base_loop (base=0x20e84a0, flags=0) at event.c:1558 #3 0x0000000000413394 in worker_libevent (arg=0x20e5fc0) at daemon/thread.c:304 #4 0x00007f6d1c4af7e1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f6d1c216ead in clone () from /lib64/libc.so.6 Thread 1 (Thread 19351): #0 0x00007f6d1c1686b5 in raise () from /lib64/libc.so.6 #1 0x00007f6d1c169e95 in abort () from /lib64/libc.so.6 #2 0x00007f6d1c161645 in __assert_fail () from /lib64/libc.so.6 #3 0x00007f6d181aeb06 in get (this=0x7f6d1a526da0, forceCreation=false, timeBound=160) at atomic.hh:109 #4 operator unsigned int (this=0x7f6d1a526da0, forceCreation=false, timeBound=160) at atomic.hh:118 #5 CheckpointManager::checkOpenCheckpoint_UNLOCKED (this=0x7f6d1a526da0, forceCreation=false, timeBound=160) at checkpoint.cc:710 #6 0x00007f6d10dfa620 in ?? () #7 0x00000000000002a3 in ?? () #8 0x00007f6d10dfa610 in ?? () #9 0x00007f6d1085b560 in ?? () #10 0x00007f6d14307500 in ?? () #11 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/membase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx 1 bin bin 31 Jun 3 08:06 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 c7fc13e0591be3155fea1efcd88c0430 /opt/membase/lib/memcached/libmemcached_utilities.so.0 /opt/membase/lib/libevent-2.0.so.5: lrwxrwxrwx 1 bin bin 21 Jun 3 08:06 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 62b1f1fede447a7642ca0f09d84eebe6 /opt/membase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx 1 root root 13 Feb 24 16:30 /lib64/libdl.so.2 -> libdl-2.12.so 643aa33f4e56d9dc502a3915703335a0 /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx 1 root root 12 Feb 24 16:30 /lib64/libm.so.6 -> libm-2.12.so b496148ca454ca96f9e39c88730b2f7a /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx 1 root root 13 Feb 24 16:30 /lib64/librt.so.1 -> librt-2.12.so d399fe4ea992cffb7227bbbc5dd827e6 /lib64/librt.so.1 /lib64/libpthread.so.0: lrwxrwxrwx 1 root root 18 Feb 24 16:30 /lib64/libpthread.so.0 -> libpthread-2.12.so e3cc21f2a81e11035cfadb97580867ef /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx 1 root root 12 Feb 24 16:30 /lib64/libc.so.6 -> libc-2.12.so b3e39b32eddb15662ba7d1c700a65bf0 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 10 Feb 24 16:30 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so bdfe53856f4614f5fb1a4fcad54d235c /lib64/ld-linux-x86-64.so.2 /opt/membase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x 1 bin bin 15322 Jun 3 03:48 /opt/membase/lib/memcached/stdin_term_handler.so 2b46627c47610208b90a34190ab97bb8 /opt/membase/lib/memcached/stdin_term_handler.so /opt/membase/lib/memcached/bucket_engine.so: lrwxrwxrwx 1 bin bin 22 Jun 3 08:06 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 bcffcfd18dcc1f8660bc8347f3eae2b3 /opt/membase/lib/memcached/bucket_engine.so /opt/membase/lib/memcached/ep.so: lrwxrwxrwx 1 bin bin 11 Jun 3 08:06 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.6.5.3_350_g2bdfd9e 1.6.5.3_350_g2bdfd9e bee1ceb429dd303b34c50756e12ba46f /opt/membase/lib/memcached/ep.so /usr/lib64/libstdc++.so.6: lrwxrwxrwx 1 root root 19 Feb 24 16:30 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 0eb10e4f59017e1796fc0b1e227b9467 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx 1 root root 28 Feb 24 16:30 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1 704445fcf4f7f80bf04c446957c31862 /lib64/libgcc_s.so.1