Basic crash dump analysis of /opt/membase/var/lib/membase/core. Please send the file to support@membase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 membase membase 5442372 2011-05-03 14:34 /opt/membase/bin/moxi -rw------- 1 membase membase 61370368 2011-05-04 00:05 /opt/membase/var/lib/membase/core d267fc252a1892969e9258b622d19c54 /opt/membase/bin/moxi 9c08bc7f59f544141c8fc91de960ae4d /opt/membase/var/lib/membase/core moxi 1.6.5.1_20_g656ca64 moxi 1.6.5.1_20_g656ca64 VERSION 1.6.5.1_20_g656ca64 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) 7.0-ubuntu Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/membase/bin/moxi...done. [New Thread 396] [New Thread 398] [New Thread 399] [New Thread 400] [New Thread 401] [New Thread 402] [New Thread 403] warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `/opt/membase/bin/moxi -B auto -z url=http://127.0.0.1:8091/pools/default/bucket'. Program terminated with signal 6, Aborted. #0 0x00007ff5fb0944b5 in raise () from /lib/libc.so.6 Thread 8 (Thread 403): #0 0x00007ff5fb132abb in read () from /lib/libc.so.6 #1 0x00007ff5fb0d2ad8 in _IO_file_underflow () from /lib/libc.so.6 #2 0x00007ff5fb0d465e in _IO_default_uflow () from /lib/libc.so.6 #3 0x00007ff5fb0cb41b in getc () from /lib/libc.so.6 #4 0x0000000000432a14 in check_stdin_thread (arg=) at stdin_check.c:16 #5 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #6 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #7 0x0000000000000000 in ?? () Thread 7 (Thread 402): #0 0x00007ff5fb1343c3 in poll () from /lib/libc.so.6 #1 0x0000000000455146 in Curl_socket_ready () #2 0x000000000046c4cd in Curl_do_perform () #3 0x000000000044e733 in run_rest_conflate (arg=0x167c2e0) at rest.c:281 #4 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #5 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (Thread 401): #0 0x00007ff5fb3db5a9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x000000000041535e in assoc_maintenance_thread (arg=) at assoc.c:220 #2 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #3 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #4 0x0000000000000000 in ?? () Thread 5 (Thread 400): #0 0x00007ff5fb140e48 in epoll_wait () from /lib/libc.so.6 #1 0x000000000044c898 in epoll_dispatch (base=0x1677300, tv=) at epoll.c:404 #2 0x0000000000446865 in event_base_loop (base=0x1677300, flags=) at event.c:1558 #3 0x0000000000416205 in worker_libevent (arg=) at thread.c:272 #4 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #5 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (Thread 399): #0 0x00007ff5fb140e48 in epoll_wait () from /lib/libc.so.6 #1 0x000000000044c898 in epoll_dispatch (base=0x1672c10, tv=) at epoll.c:404 #2 0x0000000000446865 in event_base_loop (base=0x1672c10, flags=) at event.c:1558 #3 0x0000000000416205 in worker_libevent (arg=) at thread.c:272 #4 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #5 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 3 (Thread 398): #0 0x00007ff5fb140e48 in epoll_wait () from /lib/libc.so.6 #1 0x000000000044c898 in epoll_dispatch (base=0x166e620, tv=) at epoll.c:404 #2 0x0000000000446865 in event_base_loop (base=0x166e620, flags=) at event.c:1558 #3 0x0000000000416205 in worker_libevent (arg=) at thread.c:272 #4 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #5 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #6 0x0000000000000000 in ?? () Thread 2 (Thread 396): #0 0x00007ff5fb140e48 in epoll_wait () from /lib/libc.so.6 #1 0x000000000044c898 in epoll_dispatch (base=0x16571f0, tv=) at epoll.c:404 #2 0x0000000000446865 in event_base_loop (base=0x16571f0, flags=) at event.c:1558 #3 0x0000000000409279 in main (argc=13, argv=0x7fff38cbed48) at memcached.c:5022 Thread 1 (Thread 397): #0 0x00007ff5fb0944b5 in raise () from /lib/libc.so.6 #1 0x00007ff5fb097f50 in abort () from /lib/libc.so.6 #2 0x00007ff5fb0ccc97 in ?? () from /lib/libc.so.6 #3 0x00007ff5fb158647 in __fortify_fail () from /lib/libc.so.6 #4 0x00007ff5fb1575f0 in __chk_fail () from /lib/libc.so.6 #5 0x00007ff5fb1564a7 in __strcat_chk () from /lib/libc.so.6 #6 0x00000000004061f4 in strcat (prefix=0x7ff5fb05fba0 "11220:failovertests-replica-count-1-ram-ratio-50-3d371baf-f6bc-4be7-8811-23d9c71d9d2b:pstd_stats:", name=0x49d868 "tot_downstream_connect_interval", add_stats=0x4099c0 , c=0x17a6270, fmt=0x497e1d "%llu") at /usr/include/bits/string3.h:145 #7 append_prefix_stat (prefix=0x7ff5fb05fba0 "11220:failovertests-replica-count-1-ram-ratio-50-3d371baf-f6bc-4be7-8811-23d9c71d9d2b:pstd_stats:", name=0x49d868 "tot_downstream_connect_interval", add_stats=0x4099c0 , c=0x17a6270, fmt=0x497e1d "%llu") at memcached.c:2240 #8 0x000000000043eefb in proxy_stats_dump_pstd_stats (add_stats=0x4099c0 , c=0x17a6270, pscip=0x7ff5fb05fd20) at agent_stats.c:586 #9 proxy_stats_dump_proxies (add_stats=0x4099c0 , c=0x17a6270, pscip=0x7ff5fb05fd20) at agent_stats.c:891 #10 0x000000000040ca7d in process_bin_proxy_stats (c=0x17a6270) at memcached.c:1424 #11 0x000000000042aec3 in cproxy_process_upstream_binary_nread (c=0x17a6270) at cproxy_protocol_b.c:145 #12 0x000000000040fc5b in complete_nread (c=0x17a6270) at memcached.c:1967 #13 drive_machine (c=0x17a6270) at memcached.c:3553 #14 0x0000000000446959 in event_process_active_single_queue (base=0x166a030, flags=) at event.c:1308 #15 event_process_active (base=0x166a030, flags=) at event.c:1375 #16 event_base_loop (base=0x166a030, flags=) at event.c:1572 #17 0x0000000000416205 in worker_libevent (arg=) at thread.c:272 #18 0x00007ff5fb3d6a04 in start_thread () from /lib/libpthread.so.0 #19 0x00007ff5fb14080d in clone () from /lib/libc.so.6 #20 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /lib/libz.so.1: lrwxrwxrwx 1 root root 15 2010-03-29 12:09 /lib/libz.so.1 -> libz.so.1.2.3.3 0f38b7c3dfe429aceae46c42ef08a27d /lib/libz.so.1 /usr/lib/libsasl2.so.2: lrwxrwxrwx 1 root root 18 2010-03-29 12:09 /usr/lib/libsasl2.so.2 -> libsasl2.so.2.0.23 fc7a8c9c82b260ba54f0a00e798511ed /usr/lib/libsasl2.so.2 /lib/libdl.so.2: lrwxrwxrwx 1 root root 15 2010-03-29 12:15 /lib/libdl.so.2 -> libdl-2.10.1.so ea96ff2b8fe3584f519afbcf0a3d42fe /lib/libdl.so.2 /lib/libresolv.so.2: lrwxrwxrwx 1 root root 19 2010-03-29 12:15 /lib/libresolv.so.2 -> libresolv-2.10.1.so 9409f715e7f0575f9992462b96c6cf70 /lib/libresolv.so.2 /lib/librt.so.1: lrwxrwxrwx 1 root root 15 2010-03-29 12:15 /lib/librt.so.1 -> librt-2.10.1.so 583b4d0f3b694d29b0baf4ac8b6628de /lib/librt.so.1 /lib/libm.so.6: lrwxrwxrwx 1 root root 14 2010-03-29 12:15 /lib/libm.so.6 -> libm-2.10.1.so fecb1fa2bc8c094237cc2fe88d52e43a /lib/libm.so.6 /lib/libpthread.so.0: lrwxrwxrwx 1 root root 20 2010-03-29 12:15 /lib/libpthread.so.0 -> libpthread-2.10.1.so 3db89ca924847e9b3d3b24393947a253 /lib/libpthread.so.0 /lib/libc.so.6: lrwxrwxrwx 1 root root 14 2010-03-29 12:15 /lib/libc.so.6 -> libc-2.10.1.so 392d011a2601d3bb252551fa9c8480aa /lib/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx 1 root root 12 2010-03-29 12:15 /lib64/ld-linux-x86-64.so.2 -> ld-2.10.1.so dfecf7117c592ff95dbe04e9415f1bf7 /lib64/ld-linux-x86-64.so.2 /lib/libgcc_s.so.1: -rw-r--r-- 1 root root 92552 2010-01-10 08:48 /lib/libgcc_s.so.1 439c24692746cdeb3a64601e2e5ee70b /lib/libgcc_s.so.1