Basic crash dump analysis of /data/core.memcached.23494. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rw-------. 1 couchbase couchbase 5522354176 May 23 20:47 /data/core.memcached.23494 -rwxr-xr-x. 1 bin bin 1608023 May 22 14:32 /opt/couchbase/bin/memcached 4e78e59b416d2416df6af446cab89d3d /opt/couchbase/bin/memcached 5130f9a570cf91b55a942ecfbbb17674 /data/core.memcached.23494 memcached 1.4.4_488_g71871b3 memcached 1.4.4_488_g71871b3 VERSION 1.4.4_488_g71871b3 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 23511] [New Thread 23505] [New Thread 23510] [New Thread 23506] [New Thread 23507] [New Thread 23512] [New Thread 23494] [New Thread 23514] [New Thread 23513] [New Thread 23504] [New Thread 23508] [New Thread 23503] [New Thread 23509] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 getVBucketId (this=0xcdda5c00, itm=0x0, implicitEnqueue=false) at item.hh:263 263 item.hh: No such file or directory. in item.hh Thread 13 (Thread 0x7f9c814d0700 (LWP 23509)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c8575bbe8 in epoll_dispatch (base=0x177c780, tv=) at epoll.c:404 #2 0x00007f9c85749d5a in event_base_loop (base=0x177c780, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x17548e0) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f9c852df700 (LWP 23503)): #0 0x000000366c6d840d in read () from /lib64/libc.so.6 #1 0x000000366c670f58 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x000000366c672a5e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x000000366c66df4b in getc () from /lib64/libc.so.6 #4 0x00007f9c852e0875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19 #5 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f9c81ed1700 (LWP 23508)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c8575bbe8 in epoll_dispatch (base=0x177ca00, tv=) at epoll.c:404 #2 0x00007f9c85749d5a in event_base_loop (base=0x177ca00, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x17547e8) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f9c848de700 (LWP 23504)): #0 0x000000366c6aab9d in nanosleep () from /lib64/libc.so.6 #1 0x000000366c6aaa10 in sleep () from /lib64/libc.so.6 #2 0x0000000000415c88 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f9c7e8e3700 (LWP 23513)): #0 0x000000366ca0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000366ca09328 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x000000366ca091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007f9c807c5497 in acquire (this=0x18005e8, name=Traceback (most recent call last): File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 556, in to_string header = ptr.cast(reptype) - 1 RuntimeError: Cannot access memory at address 0x80 ) at mutex.hh:70 #4 lock (this=0x18005e8, name=Traceback (most recent call last): File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 556, in to_string header = ptr.cast(reptype) - 1 RuntimeError: Cannot access memory at address 0x80 ) at locks.hh:48 #5 LockHolder (this=0x18005e8, name=Traceback (most recent call last): File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 556, in to_string header = ptr.cast(reptype) - 1 RuntimeError: Cannot access memory at address 0x80 ) at locks.hh:26 #6 TapConnMap::checkConnectivity (this=0x18005e8, name=Traceback (most recent call last): File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 556, in to_string header = ptr.cast(reptype) - 1 RuntimeError: Cannot access memory at address 0x80 ) at tapconnmap.cc:303 #7 0x00007f9c8074595a in BackFillVisitor::checkValidity (this=0x10e765e40) at backfill.cc:186 #8 0x00007f9c80747da9 in BackFillVisitor::shouldContinue (this=0x18005f0) at backfill.hh:77 #9 0x00007f9c807acded in HashTable::visit (this=0xc2721508, visitor=...) at stored-value.cc:269 #10 0x00007f9c8075ed3a in VBCBAdaptor::callback (this=0x1fd54d80, d=, t=) at ep.cc:2272 #11 0x00007f9c8075b6cf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #12 0x00007f9c8075a126 in Dispatcher::run (this=0x1796a80) at dispatcher.cc:123 #13 0x00007f9c8075a9cb in launch_dispatcher_thread (arg=0x18005f0) at dispatcher.cc:28 #14 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #15 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f9c7dee2700 (LWP 23514)): #0 0x000000366ca0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9c80787abe in wait (this=0x1800000) at syncobject.hh:42 #2 wait (this=0x1800000) at syncobject.hh:58 #3 wait (this=0x1800000) at tapconnmap.hh:171 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x1800000) at ep_engine.cc:3813 #5 0x00007f9c80787c43 in EvpNotifyPendingConns (arg=0x1800000) at ep_engine.cc:1054 #6 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #7 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f9c854e2720 (LWP 23494)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c8575bbe8 in epoll_dispatch (base=0x177c000, tv=) at epoll.c:404 #2 0x00007f9c85749d5a in event_base_loop (base=0x177c000, flags=) at event.c:1558 #3 0x0000000000408cd0 in main (argc=, argv=) at daemon/memcached.c:7504 Thread 6 (Thread 0x7f9c7f2e4700 (LWP 23512)): #0 0x000000366ca0b75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9c80757e30 in wait (this=0x1818300, d=...) at syncobject.hh:42 #2 IdleTask::run (this=0x1818300, d=...) at dispatcher.cc:286 #3 0x00007f9c8075a126 in Dispatcher::run (this=0x1796540) at dispatcher.cc:123 #4 0x00007f9c8075a9cb in launch_dispatcher_thread (arg=0x1796594) at dispatcher.cc:28 #5 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #6 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f9c828d2700 (LWP 23507)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c8575bbe8 in epoll_dispatch (base=0x177cc80, tv=) at epoll.c:404 #2 0x00007f9c85749d5a in event_base_loop (base=0x177cc80, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x17546f0) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f9c832d3700 (LWP 23506)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c8575bbe8 in epoll_dispatch (base=0x177c280, tv=) at epoll.c:404 #2 0x00007f9c85749d5a in event_base_loop (base=0x177c280, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x17545f8) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f9c80706700 (LWP 23510)): #0 0x000000366c6aab9d in nanosleep () from /lib64/libc.so.6 #1 0x000000366c6deb64 in usleep () from /lib64/libc.so.6 #2 0x00007f9c807a38a8 in updateStatsThread (arg=) at memory_tracker.cc:31 #3 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #4 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f9c83cd4700 (LWP 23505)): #0 0x000000366c6e5d03 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f9c8575bbe8 in epoll_dispatch (base=0x177c500, tv=) at epoll.c:404 #2 0x00007f9c85749d5a in event_base_loop (base=0x177c500, flags=) at event.c:1558 #3 0x0000000000413704 in worker_libevent (arg=0x1754500) at daemon/thread.c:301 #4 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x000000366c6e570d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f9c7fce5700 (LWP 23511)): #0 getVBucketId (this=0xcdda5c00, itm=0x0, implicitEnqueue=false) at item.hh:263 #1 TapProducer::completeBGFetchJob (this=0xcdda5c00, itm=0x0, implicitEnqueue=false) at tapconnection.cc:1022 #2 0x00007f9c807497a2 in TapConnMap::performTapOp (this=0x18005e8, name=, tapop=..., arg=0x0) at tapconnmap.hh:115 #3 0x00007f9c807c240b in TapBGFetchCallback::callback (this=0x49c81e50, d=..., t=std::tr1::shared_ptr (count 2144229120) 0x7f9c7fce4cc0) at tapconnection.cc:942 #4 0x00007f9c8075b6cf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #5 0x00007f9c8075a126 in Dispatcher::run (this=0x1796380) at dispatcher.cc:123 #6 0x00007f9c8075a9cb in launch_dispatcher_thread (arg=0x1) at dispatcher.cc:28 #7 0x000000366ca077f1 in start_thread () from /lib64/libpthread.so.0 #8 0x000000366c6e570d in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: lrwxrwxrwx. 1 bin bin 31 May 23 15:31 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0 403f0784777ce066f130ccb942583d7f /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 /opt/couchbase/lib/libevent-2.0.so.5: lrwxrwxrwx. 1 bin bin 21 May 23 15:31 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0 e16ee84c6cb55c4db57ebb496947f421 /opt/couchbase/lib/libevent-2.0.so.5 /lib64/libdl.so.2: lrwxrwxrwx. 1 root root 13 May 4 13:37 /lib64/libdl.so.2 -> libdl-2.12.so a67b21f1ec12dd0d73d1255e10f9ee7f /lib64/libdl.so.2 /lib64/libm.so.6: lrwxrwxrwx. 1 root root 12 May 4 13:37 /lib64/libm.so.6 -> libm-2.12.so 330c098c1cc5593706a5ecdfba8825fa /lib64/libm.so.6 /lib64/librt.so.1: lrwxrwxrwx. 1 root root 13 May 4 13:37 /lib64/librt.so.1 -> librt-2.12.so a5457240881c8f060a293426ffe1cbc7 /lib64/librt.so.1 /opt/couchbase/lib/libtcmalloc_minimal.so.4: lrwxrwxrwx. 1 bin bin 28 May 23 15:31 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0 3f3ef753c0225ae9ee373d4783451f04 /opt/couchbase/lib/libtcmalloc_minimal.so.4 /lib64/libpthread.so.0: lrwxrwxrwx. 1 root root 18 May 4 13:37 /lib64/libpthread.so.0 -> libpthread-2.12.so ec41785cddbad869a1dd1e2a1161fae2 /lib64/libpthread.so.0 /lib64/libc.so.6: lrwxrwxrwx. 1 root root 12 May 4 13:37 /lib64/libc.so.6 -> libc-2.12.so bbf3ebda4463b2499f2c6143aba88ba3 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2: lrwxrwxrwx. 1 root root 10 May 4 13:37 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so 0bbfc22a676d4c0b026c08be472e3336 /lib64/ld-linux-x86-64.so.2 /usr/lib64/libstdc++.so.6: lrwxrwxrwx. 1 root root 19 May 4 13:37 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.13 05112fb637db160bda97216c6ed64698 /usr/lib64/libstdc++.so.6 /lib64/libgcc_s.so.1: lrwxrwxrwx. 1 root root 28 May 4 13:37 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.6-20110824.so.1 3443fd6c52b3068b64501e3522c966da /lib64/libgcc_s.so.1 /opt/couchbase/lib/memcached/stdin_term_handler.so: -rwxr-xr-x. 1 bin bin 45058 May 22 14:32 /opt/couchbase/lib/memcached/stdin_term_handler.so 4e58e368155a54a87ec6c1b4c4936bc2 /opt/couchbase/lib/memcached/stdin_term_handler.so /opt/couchbase/lib/memcached/bucket_engine.so: lrwxrwxrwx. 1 bin bin 22 May 23 15:31 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0 e17374f50d649e6398651bb6f6ddbbea /opt/couchbase/lib/memcached/bucket_engine.so /opt/couchbase/lib/memcached/ep.so: lrwxrwxrwx. 1 bin bin 11 May 23 15:31 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0 EP engine v1.8.0_113_g43ebfdb 751ee6b104de2d82d6b544ba250edc3f /opt/couchbase/lib/memcached/ep.so