Basic crash dump analysis of core.memcached.29530. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x. 1 bin bin 432778 Jun 6 20:23 /opt/couchbase/bin/memcached 670a940ff485539490ad992501861704 /opt/couchbase/bin/memcached memcached 1.4.4_555_gce9ced1 memcached 1.4.4_555_gce9ced1 VERSION 1.4.4_555_gce9ced1 -rw-------. 1 couchbase couchbase 17008222208 Jun 12 15:16 core.memcached.29530 2d33242a38babe4d4ebe3784b69b0ea8 core.memcached.29530 version1.4.4_555_gce9ced1 version1.4.4_555_gce9ced1 version1.4.4_555_gce9ced1 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 29548] [New Thread 29540] [New Thread 29541] [New Thread 29543] [New Thread 29542] [New Thread 29530] [New Thread 6998] [New Thread 29546] [New Thread 29547] [New Thread 29549] [New Thread 29544] [New Thread 29535] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done. Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1 Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done. Loaded symbols for /opt/couchbase/lib/libsnappy.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 6, Aborted. #0 0x00007f85f50a1885 in raise () from /lib64/libc.so.6 Thread 12 (Thread 0x7f85f4951700 (LWP 29535)): #0 0x00007f85f51479cd in read () from /lib64/libc.so.6 #1 0x00007f85f50e0128 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00007f85f50e1c2e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00007f85f50dd11b in getc () from /lib64/libc.so.6 #4 0x00007f85f4952879 in check_stdin_thread (arg=0x403350) at extensions/daemon/stdin_check.c:19 #5 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f85f1542700 (LWP 29544)): #0 0x00007f85f51552c3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f85f5f18ba6 in epoll_dispatch (base=0x5c2e780, tv=) at epoll.c:404 #2 0x00007f85f5f07c11 in event_base_loop (base=0x5c2e780, flags=0) at event.c:1558 #3 0x0000000000413764 in worker_libevent (arg=0x17ce8e0) at daemon/thread.c:301 #4 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f85ee305700 (LWP 29549)): #0 0x00007f85f540a75b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f85efd9f5d0 in wait (this=0x5c9e080, d=...) at syncobject.hh:47 #2 IdleTask::run (this=0x5c9e080, d=...) at dispatcher.cc:294 #3 0x00007f85efda1bc0 in Dispatcher::run (this=0x5c6d880) at dispatcher.cc:123 #4 0x00007f85efda244b in launch_dispatcher_thread (arg=) at dispatcher.cc:28 #5 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f85ef707700 (LWP 29547)): #0 0x00007f85f540ddf3 in pread64 () from /lib64/libpthread.so.0 #1 0x00007f85efb275f0 in couch_pread (handle=, buf=0x1a7f8031, nbyte=8192, offset=5709824) at src/os.c:34 #2 0x00007f85efb2656a in load_buffer_from (handle=0xdd4e06f0, buf=, nbyte=8, offset=) at src/iobuffer.c:153 #3 buffered_pread (handle=0xdd4e06f0, buf=, nbyte=8, offset=) at src/iobuffer.c:291 #4 0x00007f85efb24d56 in read_skipping_prefixes (db=0x5c34a10, pos=0x7f85ef706198, len=8, dst=) at src/couch_file_read.c:25 #5 0x00007f85efb24dcd in pread_bin_internal (db=0x5c34a10, pos=5713660, ret_ptr=0x7f85ef7061e8, header=0) at src/couch_file_read.c:51 #6 0x00007f85efb24ed7 in pread_compressed (db=, pos=, ret_ptr=0x7f85ef706288) at src/couch_file_read.c:88 #7 0x00007f85efb221ce in modify_node (rq=0x7f85ef706540, nptr=0x11e3c2564, start=9, end=10, dst=0x11e3c0494) at src/btree_modify.c:296 #8 0x00007f85efb22335 in modify_node (rq=0x7f85ef706540, nptr=0x11e3c0464, start=9, end=14, dst=0x11e3be0c8) at src/btree_modify.c:443 #9 0x00007f85efb22335 in modify_node (rq=0x7f85ef706540, nptr=0x3ed0808c0, start=4, end=212, dst=0x11e3be010) at src/btree_modify.c:443 #10 0x00007f85efb22848 in modify_btree (rq=0x7f85ef706540, root=0x3ed0808c0, errcode=0x7f85ef70664c) at src/btree_modify.c:591 #11 0x00007f85efb249e7 in update_indexes (db=0x5c34a10, docs=, infos=0x2607fb258, numdocs=106, options=0) at src/couch_save.c:239 #12 couchstore_save_documents (db=0x5c34a10, docs=, infos=0x2607fb258, numdocs=106, options=0) at src/couch_save.c:369 #13 0x00007f85efe3b571 in CouchKVStore::saveDocs (this=0x5ca6000, vbid=682, rev=3, docs=0x262f11258, docinfos=0x2607fb258, docCount=106) at couch-kvstore/couch-kvstore.cc:1288 #14 0x00007f85efe3badc in CouchKVStore::commit2couchstore (this=0x5ca6000) at couch-kvstore/couch-kvstore.cc:1200 #15 0x00007f85efe3bce2 in CouchKVStore::commit (this=0x5ca6000) at couch-kvstore/couch-kvstore.cc:735 #16 0x00007f85efdb025d in TransactionContext::commit (this=0x5cc0360) at ep.cc:2653 #17 0x00007f85efdb14ed in EventuallyPersistentStore::flushSome (this=0x5cc0000, q=0x5cc02c0, rejectQueue=0x1a933c460) at ep.cc:1942 #18 0x00007f85efdd9564 in Flusher::doFlush (this=0x5c46090) at flusher.cc:244 #19 0x00007f85efdda145 in Flusher::step (this=0x5c46090, d=..., tid=std::tr1::shared_ptr (count 4) 0x5c9e300) at flusher.cc:158 #20 0x00007f85efdda7c0 in FlusherStepper::callback (this=, d=, t=) at flusher.cc:23 #21 0x00007f85efda2d0f in Task::run(Dispatcher&, std::tr1::shared_ptr) () from /opt/couchbase/lib/memcached/ep.so #22 0x00007f85efda1bc0 in Dispatcher::run (this=0x5c6cc40) at dispatcher.cc:123 #23 0x00007f85efda244b in launch_dispatcher_thread (arg=) at dispatcher.cc:28 #24 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f85f0b41700 (LWP 29546)): #0 0x00007f85f511a15d in nanosleep () from /lib64/libc.so.6 #1 0x00007f85f514e124 in usleep () from /lib64/libc.so.6 #2 0x00007f85efde0cc2 in updateStatsThread (arg=0x17cc800) at memory_tracker.cc:31 #3 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f85ecf03700 (LWP 6998)): #0 0x00007f85f511a15d in nanosleep () from /lib64/libc.so.6 #1 0x00007f85f514e124 in usleep () from /lib64/libc.so.6 #2 0x00007f85efdd9b7a in Flusher::wait (this=0x5c46090) at flusher.cc:35 #3 0x00007f85efdaf22c in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x5cc0000, __in_chrg=) at ep.cc:615 #4 0x00007f85efdd63dc in EventuallyPersistentEngine::~EventuallyPersistentEngine() () from /opt/couchbase/lib/memcached/ep.so #5 0x00007f85efdc43c5 in EvpDestroy (handle=, force=false) at ep_engine.cc:124 #6 0x00007f85f3d4a546 in engine_shutdown_thread (arg=0x5c342a0) at bucket_engine.c:1386 #7 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f85f654f720 (LWP 29530)): #0 0x00007f85f51552c3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f85f5f18ba6 in epoll_dispatch (base=0x5c2e000, tv=) at epoll.c:404 #2 0x00007f85f5f07c11 in event_base_loop (base=0x5c2e000, flags=0) at event.c:1558 #3 0x0000000000409078 in main (argc=, argv=) at daemon/memcached.c:7877 Thread 5 (Thread 0x7f85f2944700 (LWP 29542)): #0 0x00007f85f51552c3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f85f5f18ba6 in epoll_dispatch (base=0x5c2ec80, tv=) at epoll.c:404 #2 0x00007f85f5f07c11 in event_base_loop (base=0x5c2ec80, flags=0) at event.c:1558 #3 0x0000000000413764 in worker_libevent (arg=0x17ce6f0) at daemon/thread.c:301 #4 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f85f1f43700 (LWP 29543)): #0 0x00007f85f51552c3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f85f5f18ba6 in epoll_dispatch (base=0x5c2ea00, tv=) at epoll.c:404 #2 0x00007f85f5f07c11 in event_base_loop (base=0x5c2ea00, flags=0) at event.c:1558 #3 0x0000000000413764 in worker_libevent (arg=0x17ce7e8) at daemon/thread.c:301 #4 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f85f3345700 (LWP 29541)): #0 0x00007f85f51552c3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f85f5f18ba6 in epoll_dispatch (base=0x5c2e280, tv=) at epoll.c:404 #2 0x00007f85f5f07c11 in event_base_loop (base=0x5c2e280, flags=0) at event.c:1558 #3 0x0000000000413764 in worker_libevent (arg=0x17ce5f8) at daemon/thread.c:301 #4 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f85f3d46700 (LWP 29540)): #0 0x00007f85f51552c3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f85f5f18ba6 in epoll_dispatch (base=0x5c2e500, tv=) at epoll.c:404 #2 0x00007f85f5f07c11 in event_base_loop (base=0x5c2e500, flags=0) at event.c:1558 #3 0x0000000000413764 in worker_libevent (arg=0x17ce500) at daemon/thread.c:301 #4 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f85eed06700 (LWP 29548)): #0 0x00007f85f50a1885 in raise () from /lib64/libc.so.6 #1 0x00007f85f50a3065 in abort () from /lib64/libc.so.6 #2 0x00007f85efde1c50 in Mutex::acquire (this=0x5ca4000) at mutex.cc:73 #3 0x00007f85efdfeaa2 in bool TapConnMap::performTapOp(std::basic_string, std::allocator > const&, TapOperation&, Item*) () from /opt/couchbase/lib/memcached/ep.so #4 0x00007f85efdfee65 in TapBGFetchCallback::callback(Dispatcher&, std::tr1::shared_ptr) () from /opt/couchbase/lib/memcached/ep.so #5 0x00007f85efda2d0f in Task::run(Dispatcher&, std::tr1::shared_ptr) () from /opt/couchbase/lib/memcached/ep.so #6 0x00007f85efda1bc0 in Dispatcher::run (this=0x5c6ca80) at dispatcher.cc:123 #7 0x00007f85efda244b in launch_dispatcher_thread (arg=) at dispatcher.cc:28 #8 0x00007f85f54067f1 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f85f5154ccd in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: /opt/couchbase/lib/libevent-2.0.so.5: /lib64/libdl.so.2: /lib64/libm.so.6: /lib64/librt.so.1: /opt/couchbase/lib/libtcmalloc_minimal.so.4: /lib64/libpthread.so.0: /lib64/libc.so.6: /lib64/ld-linux-x86-64.so.2: /usr/lib64/libstdc++.so.6: /lib64/libgcc_s.so.1: /opt/couchbase/lib/memcached/stdin_term_handler.so: /opt/couchbase/lib/memcached/bucket_engine.so: /opt/couchbase/lib/memcached/ep.so: /opt/couchbase/lib/libcouchstore.so.1: /opt/couchbase/lib/libsnappy.so.1: /lib64/libnss_files.so.2: