Basic crash dump analysis of core.1199. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x 1 bin bin 386843 Mar 19 17:33 /opt/couchbase/./bin/memcached f9bfe052a43451e6d545911efa383f59 /opt/couchbase/./bin/memcached memcached 1.4.4_504_g3cec9b1 memcached 1.4.4_504_g3cec9b1 VERSION 1.4.4_504_g3cec9b1 -rw------- 1 couchbase couchbase 1047797760 Apr 30 21:16 core.1199 0d4cd365264b4b268babb3db57232648 core.1199 version1.4.4_504_g3cec9b1 version1.4.4_504_g3cec9b1 -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) CentOS (7.0.1-42.el5.centos) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...done. [New Thread 1214] [New Thread 1218] [New Thread 1217] [New Thread 1216] [New Thread 1215] [New Thread 1213] [New Thread 1212] [New Thread 1211] [New Thread 1210] [New Thread 1209] [New Thread 1200] [New Thread 1199] Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done. Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done. Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.0...done. Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.0 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffde3fd000 Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'. Program terminated with signal 11, Segmentation fault. #0 unlocked_find (this=0x16ed2480, vb=..., key="222befaf36627927", bucket_num=, wantDeleted=false) at stored-value.hh:168 168 stored-value.hh: No such file or directory. in stored-value.hh Thread 12 (Thread 0x2b3f76088240 (LWP 1199)): #0 0x00002b3f758f48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b3f74a81c28 in epoll_dispatch (base=0x16e24000, tv=) at epoll.c:404 #2 0x00002b3f74a70a4c in event_base_loop (base=0x16e24000, flags=0) at event.c:1558 #3 0x0000000000409956 in main (argc=15, argv=0x7fffde31a648) at daemon/memcached.c:7781 Thread 11 (Thread 1200): #0 0x00002b3f758e678b in read () from /lib64/libc.so.6 #1 0x00002b3f7588cd57 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x00002b3f7588d71e in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x00002b3f75888fdb in getc () from /lib64/libc.so.6 #4 0x00002b3f76089875 in check_stdin_thread (arg=0x403450) at extensions/daemon/stdin_check.c:19 #5 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #6 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 10 (Thread 1209): #0 0x00002b3f758ba541 in nanosleep () from /lib64/libc.so.6 #1 0x00002b3f758ba364 in sleep () from /lib64/libc.so.6 #2 0x0000000000414dd8 in check_isasl_db_thread (arg=) at daemon/isasl.c:233 #3 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #4 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 9 (Thread 1210): #0 0x00002b3f758f48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b3f74a81c28 in epoll_dispatch (base=0x16e24500, tv=) at epoll.c:404 #2 0x00002b3f74a70a4c in event_base_loop (base=0x16e24500, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x129c6500) at daemon/thread.c:304 #4 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 8 (Thread 1211): #0 0x00002b3f758f48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b3f74a81c28 in epoll_dispatch (base=0x16e24280, tv=) at epoll.c:404 #2 0x00002b3f74a70a4c in event_base_loop (base=0x16e24280, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x129c6600) at daemon/thread.c:304 #4 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 7 (Thread 1212): #0 0x00002b3f758f48a8 in epoll_wait () from /lib64/libc.so.6 #1 0x00002b3f74a81c28 in epoll_dispatch (base=0x16e24c80, tv=) at epoll.c:404 #2 0x00002b3f74a70a4c in event_base_loop (base=0x16e24c80, flags=0) at event.c:1558 #3 0x0000000000413694 in worker_libevent (arg=0x129c6700) at daemon/thread.c:304 #4 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #5 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 6 (Thread 1213): #0 0x00002b3f75612524 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b3f7560de1a in _L_lock_1034 () from /lib64/libpthread.so.0 #2 0x00002b3f7560dcdc in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000406243 in conn_add_tap_client (c=0x16da4580) at daemon/memcached.c:5800 #4 0x0000000000407226 in event_handler (fd=, which=, arg=0x16da4580) at daemon/memcached.c:5884 #5 0x00002b3f74a70df9 in event_process_active_single_queue (base=0x16e24a00, flags=0) at event.c:1308 #6 event_process_active (base=0x16e24a00, flags=0) at event.c:1375 #7 event_base_loop (base=0x16e24a00, flags=0) at event.c:1572 #8 0x0000000000413694 in worker_libevent (arg=0x129c6800) at daemon/thread.c:304 #9 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #10 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 5 (Thread 1215): #0 0x00002b3f7589c6ea in _wordcopy_fwd_aligned () from /lib64/libc.so.6 #1 0x00002b3f7589acd5 in memmove () from /lib64/libc.so.6 #2 0x00002aaaaada6cea in MemcachedEngine::processInput (this=0x16ef2800) at mc-kvstore/mc-engine.cc:970 #3 0x00002aaaaada75a8 in MemcachedEngine::maybeProcessInput (this=0x16ef2800) at mc-kvstore/mc-engine.cc:933 #4 0x00002aaaaada6489 in MemcachedEngine::sendCommand (this=0x16ef2800, rh=0x129c4f00) at mc-kvstore/mc-engine.cc:838 #5 0x00002aaaaada831d in MemcachedEngine::get (this=0x16ef2800, key="78686c3c215f5442", vb=, cb=...) at mc-kvstore/mc-engine.cc:1128 #6 0x00002aaaaadabc74 in MCKVStore::get (this=0x129c4b80, key="78686c3c215f5442", vb=65, cb=...) at mc-kvstore/mc-kvstore.cc:57 #7 0x00002aaaaad8cf98 in TapBGFetchCallback::callback (this=0x16e31860, d=, t=std::tr1::shared_ptr (count 0) 0x454d5f50) at tapconnection.cc:957 #8 0x00002aaaaad1daaf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #9 0x00002aaaaad1cc01 in Dispatcher::run (this=0x16ecc1c0) at dispatcher.cc:119 #10 0x00002aaaaad1d453 in launch_dispatcher_thread (arg=0x19e716b0) at dispatcher.cc:28 #11 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #12 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 4 (Thread 1216): #0 0x00002b3f758eb696 in poll () from /lib64/libc.so.6 #1 0x00002aaaaada69f3 in MemcachedEngine::waitForReadable (this=0x16ef2800) at mc-kvstore/mc-engine.cc:1015 #2 0x00002aaaaada6db9 in MemcachedEngine::wait (this=0x45ed6a80) at mc-kvstore/mc-engine.cc:1048 #3 0x00002aaaaada7516 in MemcachedEngine::tap (this=0x16ef2800, cb=std::tr1::shared_ptr (count -1425288096) 0x45ed6bb0) at mc-kvstore/mc-engine.cc:1284 #4 0x00002aaaaadacf65 in MCKVStore::dump (this=0x129c4b80, cb=std::tr1::shared_ptr (count 0) 0x45ed6ca0) at mc-kvstore/mc-kvstore.cc:150 #5 0x00002aaaaad258d5 in EventuallyPersistentStore::warmup (this=0x16ed2480, st=std::map with 103 elements = {...}, keysOnly=false) at ep.cc:2682 #6 0x00002aaaaad61cb1 in Flusher::step (this=0x16ed05a0, d=, tid=Traceback (most recent call last): File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string return '%s (count %d) %s' % (self.typename, TypeError: int argument required ) at flusher.cc:211 #7 0x00002aaaaad6236e in FlusherStepper::callback (this=0x348a9400, d=..., t=) at flusher.cc:23 #8 0x00002aaaaad1daaf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #9 0x00002aaaaad1cc01 in Dispatcher::run (this=0x16ecc380) at dispatcher.cc:119 #10 0x00002aaaaad1d453 in launch_dispatcher_thread (arg=0x45ed6a80) at dispatcher.cc:28 #11 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #12 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 3 (Thread 1217): #0 0x00002aaaaad76385 in HashTable::visit (this=0x17020c08, visitor=...) at stored-value.hh:1357 #1 0x00002aaaaad21199 in VBCBAdaptor::callback (this=0x16f16090, d=, t=Traceback (most recent call last): File "/usr/share/gdb/python/libstdcxx/v6/printers.py", line 32, in to_string return '%s (count %d) %s' % (self.typename, TypeError: int argument required ) at ep.cc:2922 #2 0x00002aaaaad1daaf in Task::run (this=, d=..., t=) at dispatcher.hh:139 #3 0x00002aaaaad1cc01 in Dispatcher::run (this=0x16ecc8c0) at dispatcher.cc:119 #4 0x00002aaaaad1d453 in launch_dispatcher_thread (arg=0x19b7d486) at dispatcher.cc:28 #5 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #6 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 2 (Thread 1218): #0 0x00002b3f75610150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002aaaaad43700 in wait (this=0x16ed6000) at syncobject.hh:42 #2 wait (this=0x16ed6000) at syncobject.hh:58 #3 wait (this=0x16ed6000) at tapconnmap.hh:209 #4 EventuallyPersistentEngine::notifyPendingConnections (this=0x16ed6000) at ep_engine.cc:3197 #5 0x00002aaaaad437d1 in EvpNotifyPendingConns (arg=0x16ed6000) at ep_engine.cc:971 #6 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #7 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x44ad5940 (LWP 1214)): #0 unlocked_find (this=0x16ed2480, vb=..., key="222befaf36627927", bucket_num=, wantDeleted=false) at stored-value.hh:168 #1 EventuallyPersistentStore::fetchValidValue (this=0x16ed2480, vb=..., key="222befaf36627927", bucket_num=, wantDeleted=false) at ep.cc:834 #2 0x00002aaaaad2dd58 in EventuallyPersistentStore::getInternal(const std::locale::string &, uint16_t, const void *, bool, bool, ._101) (this=0x16ed2480, key="222befaf36627927", vbucket=, cookie=0x16da42c0, queueBG=false, honorStates=false, allowedState=vbucket_state_active) at ep.cc:1348 #3 0x00002aaaaad87d40 in TapProducer::getNextItem(const void *, uint16_t *, ._174 &) (this=0x1c018380, c=0x16da42c0, vbucket=0x44ad47b8, ret=@0x44ad067c) at ep.hh:469 #4 0x00002aaaaad55e49 in doWalkTapQueue (this=0x16ed6000, cookie=0x16da42c0, itm=0x44ad47a8, es=0x44ad47a0, nes=0x44ad47bc, ttl=0x44ad47bf "\377@P\255D", flags=0x44ad47ba, seqno=0x44ad47b4, vbucket=0x44ad47b8) at ep_engine.cc:1446 #5 EventuallyPersistentEngine::walkTapQueue (this=0x16ed6000, cookie=0x16da42c0, itm=0x44ad47a8, es=0x44ad47a0, nes=0x44ad47bc, ttl=0x44ad47bf "\377@P\255D", flags=0x44ad47ba, seqno=0x44ad47b4, vbucket=0x44ad47b8) at ep_engine.cc:1504 #6 0x00002aaaaaaaf696 in bucket_tap_iterator_shim (handle=0x2aaaaacb5480, cookie=0x16da42c0, itm=0x44ad47a8, engine_specific=0x44ad47a0, nengine_specific=0x44ad47bc, ttl=0x44ad47bf "\377@P\255D", flags=0x44ad47ba, seqno=0x44ad47b4, vbucket=0x44ad47b8) at bucket_engine.c:2041 #7 0x000000000040f892 in ship_tap_log (c=0x16da42c0) at daemon/memcached.c:2596 #8 conn_ship_log (c=0x16da42c0) at daemon/memcached.c:5430 #9 0x0000000000407234 in event_handler (fd=, which=, arg=0x16da42c0) at daemon/memcached.c:5884 #10 0x00002b3f74a70df9 in event_process_active_single_queue (base=0x16e24780, flags=0) at event.c:1308 #11 event_process_active (base=0x16e24780, flags=0) at event.c:1375 #12 event_base_loop (base=0x16e24780, flags=0) at event.c:1572 #13 0x0000000000413694 in worker_libevent (arg=0x129c6900) at daemon/thread.c:304 #14 0x00002b3f7560b73d in start_thread () from /lib64/libpthread.so.0 #15 0x00002b3f758f44bd in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: /opt/couchbase/lib/memcached/libmemcached_utilities.so.0: /opt/couchbase/lib/libevent-2.0.so.5: /lib64/libdl.so.2: /lib64/libm.so.6: /lib64/librt.so.1: /opt/couchbase/lib/libtcmalloc_minimal.so.0: /lib64/libpthread.so.0: /lib64/libc.so.6: /lib64/ld-linux-x86-64.so.2: /usr/lib64/libstdc++.so.6: /lib64/libgcc_s.so.1: /opt/couchbase/lib/memcached/stdin_term_handler.so: /opt/couchbase/lib/memcached/bucket_engine.so: /opt/couchbase/lib/memcached/ep.so: /lib64/libnss_files.so.2: