Basic crash dump analysis of core.4713. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 386843 Mar 19 14:33 /opt/couchbase/./bin/memcached f9bfe052a43451e6d545911efa383f59 /opt/couchbase/./bin/memcached memcached 1.4.4_504_g3cec9b1 memcached 1.4.4_504_g3cec9b1 VERSION 1.4.4_504_g3cec9b1 -rw------- 1 couchbase couchbase 610848768 Apr 27 11:33 core.4713 1fb6fa4182e395c2c870d25646926c32 core.4713 version1.4.4_504_g3cec9b1 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) CentOS (7.0.1-42.el5.centos) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 4730] [New Thread 4732] [New Thread 4731] [New Thread 4729] [New Thread 4728] [New Thread 4727] [New Thread 4726] [New Thread 4725] [New Thread 4724] [New Thread 4723] [New Thread 4722] [New Thread 4713] Error while mapping shared library sections: ":"edc2@21.com","city":"1e1","country":"f4","realm":"08","coins":608.660000,"achievements":[28,60,60,140,217,45,240],"body":"xc4ca4238a0b923820dcc509a6f7: No such file or directory. Error while mapping shared library sections: cc509a6f7: No such file or directory. Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Symbol file not found for ":"edc2@21.com","city":"1e1","country":"f4","realm":"08","coins":608.660000,"achievements":[28,60,60,140,217,45,240],"body":"xc4ca4238a0b923820dcc509a6f7 Symbol file not found for cc509a6f7 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff8cffd000 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 tcmalloc::CentralFreeList::FetchFromSpans (this=0x2b100cceb6e0) at src/central_freelist.cc:298 298 src/central_freelist.cc: No such file or directory. in src/central_freelist.cc Thread 12 (Thread 0x2b100cd8c220 (LWP 4713)): #0 0x00000035190d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b100c889c28 in epoll_dispatch (base=0x20c66000, tv=) at epoll.c:404 #2 0x00002b100c878a4c in event_base_loop (base=0x20c66000, flags=0) at event.c:1558 #3 0x0000000000409956 in main (argc=15, argv=0x7fff8cfdab68) at daemon/memcached.c:7781 Thread 11 (Thread 4722): #0 0x00000035190c678b in read () from /lib64/libc.so.6 #1 0x000000351906cd57 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x000000351906d71e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x0000003519068fdb in getc () from /lib64/libc.so.6 #4 0x00002b100cd8d875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19 #5 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #6 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 10 (Thread 4723): #0 0x000000351909a541 in nanosleep () from /lib64/libc.so.6 #1 0x000000351909a364 in sleep () from /lib64/libc.so.6 #2 0x0000000000414dd8 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #4 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 9 (Thread 4724): #0 0x00000035190d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b100c889c28 in epoll_dispatch (base=0x20c66500, tv=) at epoll.c:404 #2 0x00002b100c878a4c in event_base_loop (base=0x20c66500, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x1c808500) at daemon/thread.c:304 #4 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 8 (Thread 4725): #0 0x00000035190d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b100c889c28 in epoll_dispatch (base=0x20c66280, tv=) at epoll.c:404 #2 0x00002b100c878a4c in event_base_loop (base=0x20c66280, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x1c808600) at daemon/thread.c:304 #4 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 7 (Thread 4726): #0 0x00000035190d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b100c889c28 in epoll_dispatch (base=0x20c66c80, tv=) at epoll.c:404 #2 0x00002b100c878a4c in event_base_loop (base=0x20c66c80, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x1c808700) at daemon/thread.c:304 #4 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 6 (Thread 4727): #0 0x00000035190d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b100c889c28 in epoll_dispatch (base=0x20c66a00, tv=) at epoll.c:404 #2 0x00002b100c878a4c in event_base_loop (base=0x20c66a00, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x1c808800) at daemon/thread.c:304 #4 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 5 (Thread 4728): #0 0x00000035190d48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b100c889c28 in epoll_dispatch (base=0x20c66780, tv=) at epoll.c:404 #2 0x00002b100c878a4c in event_base_loop (base=0x20c66780, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x1c808900) at daemon/thread.c:304 #4 0x0000003519c0673d in start_thread () from /lib64/libpthread.so.0 #5 0x00000035190d44bd in clone () from /lib64/libc.so.6 Thread 4 (Thread 4729): #0 0x000000351907cab5 in _wordcopy_bwd_dest_aligned () from /lib64/libc.so.6 #1 0x000000351907adbb in memmove () from /lib64/libc.so.6 #2 0x00002aaaaada6cea in ?? () #3 0x0000000020d36870 in ?? () #4 0x0000000020d34800 in ?? () #5 0x000000001c806f00 in ?? () #6 0x0000000020c72f18 in ?? () #7 0x0000000000000000 in ?? () Thread 3 (Thread 4731): #0 0x0000003519c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad1adb8 in ?? () #2 0x000000004f9ae661 in ?? () #3 0x00000000197c94f8 in ?? () #4 0x0000000046c70050 in ?? () #5 0x00002aaaaad6f902 in ?? () #6 0x0000000020d2a080 in ?? () #7 0x00002aaaaad1ad30 in ?? () #8 0x0000000020d0e8c0 in ?? () #9 0x0000000020d2a080 in ?? () #10 0x0000000000000002 in ?? () #11 0x00002aaaaad1cc01 in ?? () #12 0x0000000046c6ffb0 in ?? () #13 0x0000000020d0e8d0 in ?? () #14 0x0000000000000000 in ?? () Thread 2 (Thread 4732): #0 0x0000003519c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad43700 in ?? () #2 0x000000004f9ae661 in ?? () #3 0x00000000078d6920 in ?? () #4 0x000000004f9ae661 in ?? () #5 0x000000000001eef4 in ?? () #6 0x0000000000000000 in ?? () Thread 1 (Thread 0x4626f940 (LWP 4730)): #0 tcmalloc::CentralFreeList::FetchFromSpans (this=0x2b100cceb6e0) at src/central_freelist.cc:298 #1 0x00002b100cacea49 in tcmalloc::CentralFreeList::FetchFromSpansSafe (this=0x2b100cceb6e0) at src/central_freelist.cc:283 #2 0x00002b100caceae9 in tcmalloc::CentralFreeList::RemoveRange (this=0x2b100cceb6e0, start=0x4626ec10, end=0x4626ec08, N=) at src/central_freelist.cc:263 #3 0x00002b100cad23a7 in tcmalloc::ThreadCache::FetchFromCentralCache (this=0x1c7e2a80, cl=, byte_size=112) at src/thread_cache.cc:156 #4 0x00002b100cacadef in Allocate (size=106, nothrow=false) at src/thread_cache.h:342 #5 do_malloc (size=106, nothrow=false) at src/tcmalloc.cc:1009 #6 (anonymous namespace)::cpp_alloc (size=106, nothrow=false) at src/tcmalloc.cc:1287 #7 0x00002b100cad7e50 in tc_new (size=47347934344928) at src/tcmalloc.cc:1463 #8 0x000000352ba9b861 in std::basic_string, std::allocator >::_Rep::_S_create(unsigned long, unsigned long, std::allocator const&) () from /usr/lib64/libstdc++.so.6 #9 0x00002aaaaad0ab55 in ?? () #10 0x0000000020d125a0 in ?? () #11 0x0000000020c6e328 in ?? () #12 0x000000352bcefe00 in ?? () from /usr/lib64/libstdc++.so.6 #13 0x00002b100c665d50 in ?? () at utilities/extension_loggers.c:87 from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 #14 0x0000000039e9e0d0 in ?? () #15 0x00002aaaaad61fc0 in ?? () #16 0x0000000020d40028 in ?? () #17 0x00002aaaaada4b23 in ?? () #18 0x0000000020fe9110 in ?? () #19 0x0000003519c0dcf4 in recv () from /lib64/libpthread.so.0 #20 0x0000000039a12258 in ?? () #21 0x0000000039a12258 in ?? () #22 0x0000000039a12258 in ?? () #23 0x0000000039a122a9 in ?? () #24 0x0000000039a12458 in ?? () #25 0x000000352bced720 in ?? () from /usr/lib64/libstdc++.so.6 #26 0x00002aaa00000018 in ?? () #27 0x0000000039a12258 in ?? () #28 0x000000352bce92c8 in vtable for std::basic_stringstream, std::allocator > () from /usr/lib64/libstdc++.so.6 #29 0x0000000000000006 in ?? () #30 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: /opt/couchbase/lib/libevent-2.0.so.5: /lib64/libdl.so.2: /lib64/libm.so.6: /lib64/librt.so.1: /opt/couchbase/lib/libtcmalloc_minimal.so.0: /lib64/libpthread.so.0: /lib64/libc.so.6: /lib64/ld-linux-x86-64.so.2: /usr/lib64/libstdc++.so.6: /lib64/libgcc_s.so.1: /opt/couchbase/lib/memcached/stdin_term_handler.so: