Basic crash dump analysis of /tmp/core.memcached.19334.

Please send the file to support@membase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin     bin        392488 Jun  3 03:48 /opt/membase/bin/memcached
-rw------- 1 membase membase 458076160 Jun  3 06:36 /tmp/core.memcached.19334
b913e0e184128d9354565e7ae7ddf56e  /opt/membase/bin/memcached
856da1210a00655d1c03ed903e0b957e  /tmp/core.memcached.19334
memcached 1.4.4_461_gf99c147
memcached 1.4.4_461_gf99c147
VERSION 1.4.4_461_gf99c147
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) Amazon Linux (7.1-29.9.amzn1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/membase/bin/memcached...done.
[New Thread 19343]
[New Thread 19355]
[New Thread 19354]
[New Thread 19356]
[New Thread 19340]
[New Thread 19341]
[New Thread 19334]
[New Thread 19345]
[New Thread 19347]
[New Thread 19357]
[New Thread 19344]
[New Thread 19346]

warning: .dynamic section for "/opt/membase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /opt/membase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/membase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/membase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/membase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/membase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/membase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/membase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/membase/lib/memcached/bucket_engine.so
Reading symbols from /opt/membase/lib/memcached/ep.so...done.
Loaded symbols for /opt/membase/lib/memcached/ep.so
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Core was generated by `/opt/membase/bin/memcached -X /opt/membase/lib/memcached/stdin_term_handler.so'.
Program terminated with signal 6, Aborted.
#0  0x00007fde93ca16b5 in raise () from /lib64/libc.so.6

Thread 12 (Thread 19346):
#0  0x00007fde93d504a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fde948afc28 in epoll_dispatch (base=0x23cdb00, tv=<value optimized out>) at epoll.c:404
#2  0x00007fde9489ea4c in event_base_loop (base=0x23cdb00, flags=0) at event.c:1558
#3  0x0000000000413394 in worker_libevent (arg=0x23cbec0) at daemon/thread.c:304
#4  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 11 (Thread 19344):
#0  0x00007fde93d504a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fde948afc28 in epoll_dispatch (base=0x23cc9c0, tv=<value optimized out>) at epoll.c:404
#2  0x00007fde9489ea4c in event_base_loop (base=0x23cc9c0, flags=0) at event.c:1558
#3  0x0000000000413394 in worker_libevent (arg=0x23cbcc0) at daemon/thread.c:304
#4  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 10 (Thread 19357):
#0  0x00007fde93fec7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fde7bcb1f37 in release (this=0x7fde8c036d60, when=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/boost_shared_ptr.h:153
#2  ~shared_count (this=0x7fde8c036d60, when=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/boost_shared_ptr.h:277
#3  ~shared_ptr (this=0x7fde8c036d60, when=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/boost_shared_ptr.h:486
#4  EventuallyPersistentEngine::flush (this=0x7fde8c036d60, when=<value optimized out>) at ep_engine.cc:1847
#5  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 9 (Thread 19347):
#0  0x00007fde93d504a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fde948afc28 in epoll_dispatch (base=0x23ce4a0, tv=<value optimized out>) at epoll.c:404
#2  0x00007fde9489ea4c in event_base_loop (base=0x23ce4a0, flags=0) at event.c:1558
#3  0x0000000000413394 in worker_libevent (arg=0x23cbfc0) at daemon/thread.c:304
#4  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 8 (Thread 19345):
#0  0x00007fde93d504a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fde948afc28 in epoll_dispatch (base=0x23cd260, tv=<value optimized out>) at epoll.c:404
#2  0x00007fde9489ea4c in event_base_loop (base=0x23cd260, flags=0) at event.c:1558
#3  0x0000000000413394 in worker_libevent (arg=0x23cbdc0) at daemon/thread.c:304
#4  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 7 (Thread 19334):
#0  0x00007fde93d504a3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007fde948afc28 in epoll_dispatch (base=0x23c0b10, tv=<value optimized out>) at epoll.c:404
#2  0x00007fde9489ea4c in event_base_loop (base=0x23c0b10, flags=0) at event.c:1558
#3  0x0000000000409974 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7459

Thread 6 (Thread 19341):
#0  0x00007fde93d1546d in nanosleep () from /lib64/libc.so.6
#1  0x00007fde93d152e0 in sleep () from /lib64/libc.so.6
#2  0x0000000000414f58 in check_isasl_db_thread (arg=<value optimized out>) at daemon/isasl.c:233
#3  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 5 (Thread 19340):
#0  0x00007fde93d42b5d in read () from /lib64/libc.so.6
#1  0x00007fde93ce0228 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x00007fde93ce1d2e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3  0x00007fde93cdd24b in getc () from /lib64/libc.so.6
#4  0x00007fde93a6e855 in check_stdin_thread (arg=0x4034b0) at extensions/daemon/stdin_check.c:19
#5  0x00007fde93fe87e1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fde93d4fead in clone () from /lib64/libc.so.6

Thread 4 (Thread 19356):
#0  0x00007fde93fec7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fde7bc906b0 in Dispatcher::start (this=0x7fde8e017804) at dispatcher.cc:39
#2  0x000000004de880f5 in ?? ()
#3  0x000000000785ef10 in ?? ()
#4  0x00007fde7a728df0 in ?? ()
#5  0x00007fde7bc8ce27 in ~pair (this=0x7fde7a729700, k=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_pair.h:69
#6  Internal::map_base<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry>, Internal::extract1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry> >, true, std::tr1::hashtable<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry>, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry> >, Internal::extract1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry> >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, Internal::mod_range_hashing, Internal::default_ranged_hash, Internal::prime_rehash_policy, false, false, true> >::operator[] (this=0x7fde7a729700, k=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/hashtable:691
#7  0x0000000000000000 in ?? ()

Thread 3 (Thread 19354):
#0  0x00007fde93fec43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fde7bc92823 in Dispatcher::completeNonDaemonTasks (this=0x7fde7b72b700) at dispatcher.cc:234
#2  0x0000000000000000 in ?? ()

Thread 2 (Thread 19355):
#0  0x00007fde93fec7a9 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fde7bc906b0 in Dispatcher::start (this=0x7fde8d0d5324) at dispatcher.cc:39
#2  0x000000004de880f2 in ?? ()
#3  0x000000000d8c0480 in ?? ()
#4  0x00007fde7af29df0 in ?? ()
#5  0x00007fde7bc8ce27 in ~pair (this=0x7fde7af2a700, k=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_pair.h:69
#6  Internal::map_base<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry>, Internal::extract1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry> >, true, std::tr1::hashtable<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry>, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry> >, Internal::extract1st<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, index_entry> >, std::equal_to<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::tr1::hash<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, Internal::mod_range_hashing, Internal::default_ranged_hash, Internal::prime_rehash_policy, false, false, true> >::operator[] (this=0x7fde7af2a700, k=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/tr1/hashtable:691
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 19343):
#0  0x00007fde93ca16b5 in raise () from /lib64/libc.so.6
#1  0x00007fde93ca2e95 in abort () from /lib64/libc.so.6
#2  0x00007fde93c9a645 in __assert_fail () from /lib64/libc.so.6
#3  0x00007fde7bc89b06 in get (this=0x7fde92860da0, forceCreation=false, timeBound=48) at atomic.hh:109
#4  operator unsigned int (this=0x7fde92860da0, forceCreation=false, timeBound=48) at atomic.hh:118
#5  CheckpointManager::checkOpenCheckpoint_UNLOCKED (this=0x7fde92860da0, forceCreation=false, timeBound=48) at checkpoint.cc:710
#6  0x00007fde8ed2fff0 in ?? ()
#7  0x00000000000003f9 in ?? ()
#8  0x00007fde8ed2ffe0 in ?? ()
#9  0x00007fde8ed2ff20 in ?? ()
#10 0x00007fde8c307500 in ?? ()
#11 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information:
/opt/membase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Jun  3 08:06 /opt/membase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
c7fc13e0591be3155fea1efcd88c0430  /opt/membase/lib/memcached/libmemcached_utilities.so.0
/opt/membase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Jun  3 08:06 /opt/membase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
62b1f1fede447a7642ca0f09d84eebe6  /opt/membase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 13 Feb 24 16:30 /lib64/libdl.so.2 -> libdl-2.12.so
643aa33f4e56d9dc502a3915703335a0  /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 12 Feb 24 16:30 /lib64/libm.so.6 -> libm-2.12.so
b496148ca454ca96f9e39c88730b2f7a  /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 13 Feb 24 16:30 /lib64/librt.so.1 -> librt-2.12.so
d399fe4ea992cffb7227bbbc5dd827e6  /lib64/librt.so.1
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 18 Feb 24 16:30 /lib64/libpthread.so.0 -> libpthread-2.12.so
e3cc21f2a81e11035cfadb97580867ef  /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 12 Feb 24 16:30 /lib64/libc.so.6 -> libc-2.12.so
b3e39b32eddb15662ba7d1c700a65bf0  /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 10 Feb 24 16:30 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so
bdfe53856f4614f5fb1a4fcad54d235c  /lib64/ld-linux-x86-64.so.2
/opt/membase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 15322 Jun  3 03:48 /opt/membase/lib/memcached/stdin_term_handler.so
2b46627c47610208b90a34190ab97bb8  /opt/membase/lib/memcached/stdin_term_handler.so
/opt/membase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Jun  3 08:06 /opt/membase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
bcffcfd18dcc1f8660bc8347f3eae2b3  /opt/membase/lib/memcached/bucket_engine.so
/opt/membase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Jun  3 08:06 /opt/membase/lib/memcached/ep.so -> ep.so.0.0.0
EP engine v1.6.5.3_350_g2bdfd9e
1.6.5.3_350_g2bdfd9e
bee1ceb429dd303b34c50756e12ba46f  /opt/membase/lib/memcached/ep.so
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 Feb 24 16:30 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13
0eb10e4f59017e1796fc0b1e227b9467  /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Feb 24 16:30 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.4-20100726.so.1
704445fcf4f7f80bf04c446957c31862  /lib64/libgcc_s.so.1