Basic crash dump analysis of /data/core.memcached.15739. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rw-------. 1 couchbase couchbase 5462253568 May 23 20:11 /data/core.memcached.15739 -rwxr-xr-x. 1 bin bin 1608023 May 22 14:32 /opt/couchbase/bin/memcached 4e78e59b416d2416df6af446cab89d3d /opt/couchbase/bin/memcached 22afbb3b26a3279a1d706fd7bec0eda2 /data/core.memcached.15739 memcached 1.4.4_488_g71871b3 memcached 1.4.4_488_g71871b3 VERSION 1.4.4_488_g71871b3 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 15756] [New Thread 15757] [New Thread 15758] [New Thread 15739] [New Thread 15753] [New Thread 15749] [New Thread 15750] [New Thread 15759] [New Thread 15752] [New Thread 15751] [New Thread 15755] [New Thread 15754] [New Thread 15748] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 getVBucketId (this=0x50d8dc00, itm=0x0, implicitEnqueue=false) at item.hh:263 263 item.hh: No such file or directory. in item.hh Thread 13 (Thread 0x7f19f211f700 (LWP 15748)): #0 0x000000366c6d840d in read () from /lib64/libc.so.6 #1 0x000000366c670f58 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x000000366c672a5e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x000000366c66df4b in getc () from /lib64/libc.so.6 #4 0x00007f19f2120875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19 #5 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f19ee310700 (LWP 15754)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f19f259bbe8 in epoll_dispatch (base=0x10d0780, tv=) at epoll.c:404 #2 0x00007f19f2589d5a in event_base_loop (base=0x10d0780, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x10a88e0) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f19ed546700 (LWP 15755)): #0 0x000000366c6aab9d in nanosleep () from /lib64/libc.so.6 #1 0x000000366c6deb64 in usleep () from /lib64/libc.so.6 #2 0x00007f19ed5e38a8 in updateStatsThread (arg=) at memory_tracker.cc:31 #3 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f19f0113700 (LWP 15751)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f19f259bbe8 in epoll_dispatch (base=0x10d0280, tv=) at epoll.c:404 #2 0x00007f19f2589d5a in event_base_loop (base=0x10d0280, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x10a85f8) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f19ef712700 (LWP 15752)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f19f259bbe8 in epoll_dispatch (base=0x10d0c80, tv=) at epoll.c:404 #2 0x00007f19f2589d5a in event_base_loop (base=0x10d0c80, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x10a86f0) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f19ead22700 (LWP 15759)): #0 0x000000366ca0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f19ed5c7abe in wait (this=0x1154000) at syncobject.hh:42 #2 wait (this=0x1154000) at syncobject.hh:58 #3 wait (this=0x1154000) at tapconnmap.hh:171 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x1154000) at ep_engine.cc:3813 #5 0x00007f19ed5c7c43 in EvpNotifyPendingConns (arg=0x1154000) at ep_engine.cc:1054 #6 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f19f0b14700 (LWP 15750)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f19f259bbe8 in epoll_dispatch (base=0x10d0500, tv=) at epoll.c:404 #2 0x00007f19f2589d5a in event_base_loop (base=0x10d0500, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x10a8500) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f19f171e700 (LWP 15749)): #0 0x000000366c6aab9d in nanosleep () from /lib64/libc.so.6 #1 0x000000366c6aaa10 in sleep () from /lib64/libc.so.6 #2 0x0000000000415c88 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f19eed11700 (LWP 15753)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f19f259bbe8 in epoll_dispatch (base=0x10d0a00, tv=) at epoll.c:404 #2 0x00007f19f2589d5a in event_base_loop (base=0x10d0a00, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x10a87e8) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f19f2322720 (LWP 15739)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f19f259bbe8 in epoll_dispatch (base=0x10d0000, tv=) at epoll.c:404 #2 0x00007f19f2589d5a in event_base_loop (base=0x10d0000, flags=) at event.c:1558 #3 0x0000000000408cd0 in main (argc=, argv=) at daemon/memcached.c:7504 Thread 3 (Thread 0x7f19eb723700 (LWP 15758)): #0 0x000000366ca0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f19ed597e30 in wait (this=0x116c980, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x116c980, d=...) at dispatcher.cc:286 #3 0x00007f19ed59a126 in Dispatcher::run (this=0x10eaa80) at dispatcher.cc:123 #4 0x00007f19ed59a9cb in launch_dispatcher_thread (arg=0x10eaad4) at dispatcher.cc:28 #5 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f19ec124700 (LWP 15757)): #0 0x000000366ca0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f19ed597e30 in wait (this=0x116c300, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x116c300, d=...) at dispatcher.cc:286 #3 0x00007f19ed59a126 in Dispatcher::run (this=0x10ea540) at dispatcher.cc:123 #4 0x00007f19ed59a9cb in launch_dispatcher_thread (arg=0x10ea594) at dispatcher.cc:28 #5 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f19ecb25700 (LWP 15756)): #0 getVBucketId (this=0x50d8dc00, itm=0x0, implicitEnqueue=false) at item.hh:263 #1 TapProducer::completeBGFetchJob (this=0x50d8dc00, itm=0x0, implicitEnqueue=false) at tapconnection.cc:1022 #2 0x00007f19ed5897a2 in TapConnMap::performTapOp (this=0x11545e8, name=, tapop=..., arg=0x0) at tapconnmap.hh:115 #3 0x00007f19ed60240b in TapBGFetchCallback::callback (this=0x6d8a190, d=..., t=std::tr1::shared_ptr (count -323856640) 0x7f19ecb24cc0) at tapconnection.cc:942 #4 0x00007f19ed59b6cf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #5 0x00007f19ed59a126 in Dispatcher::run (this=0x10ea380) at dispatcher.cc:123 #6 0x00007f19ed59a9cb in launch_dispatcher_thread (arg=0x1) at dispatcher.cc:28 #7 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #8 0x000000366c6e570d in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx. 1 bin bin 31 May 23 15:31 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 403f0784777ce066f130ccb942583d7f /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 /opt/couchbase/lib/libevent-2.0.so.5: lrwxrwxrwx. 1 bin bin 21 May 23 15:31 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 e16ee84c6cb55c4db57ebb496947f421 /opt/couchbase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx. 1 root root 13 May 4 13:37 /lib64/libdl.so.2 -> libdl-2.12.so a67b21f1ec12dd0d73d1255e10f9ee7f /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx. 1 root root 12 May 4 13:37 /lib64/libm.so.6 -> libm-2.12.so 330c098c1cc5593706a5ecdfba8825fa /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx. 1 root root 13 May 4 13:37 /lib64/librt.so.1 -> librt-2.12.so a5457240881c8f060a293426ffe1cbc7 /lib64/librt.so.1 /opt/couchbase/lib/libtcmalloc_minimal.so.4: lrwxrwxrwx. 1 bin bin 28 May 23 15:31 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0 3f3ef753c0225ae9ee373d4783451f04 /opt/couchbase/lib/libtcmalloc_minimal.so.4 /lib64/libpthread.so.0: lrwxrwxrwx. 1 root root 18 May 4 13:37 /lib64/libpthread.so.0 -> libpthread-2.12.so ec41785cddbad869a1dd1e2a1161fae2 /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx. 1 root root 12 May 4 13:37 /lib64/libc.so.6 -> libc-2.12.so bbf3ebda4463b2499f2c6143aba88ba3 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx. 1 root root 10 May 4 13:37 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so 0bbfc22a676d4c0b026c08be472e3336 /lib64/ld-linux-x86-64.so.2 /usr/lib64/libstdc++.so.6: lrwxrwxrwx. 1 root root 19 May 4 13:37 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 05112fb637db160bda97216c6ed64698 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx. 1 root root 28 May 4 13:37 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.6-20110824.so.1 3443fd6c52b3068b64501e3522c966da /lib64/libgcc_s.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x. 1 bin bin 45058 May 22 14:32 /opt/couchbase/lib/memcached/stdin_term_handler.so 4e58e368155a54a87ec6c1b4c4936bc2 /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/bucket_engine.so: lrwxrwxrwx. 1 bin bin 22 May 23 15:31 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 e17374f50d649e6398651bb6f6ddbbea /opt/couchbase/lib/memcached/bucket_engine.so /opt/couchbase/lib/memcached/ep.so: lrwxrwxrwx. 1 bin bin 11 May 23 15:31 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.8.0_113_g43ebfdb 751ee6b104de2d82d6b544ba250edc3f /opt/couchbase/lib/memcached/ep.so