dhaikney@cb-server-1:~$ sudo gdb attach 31694 GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: ... attach: No such file or directory. Attaching to process 31694 Reading symbols from /opt/couchbase/bin/memcached...done. Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done. Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done. Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done. Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done. Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 Reading symbols from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 Reading symbols from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7f0d4ca82700 (LWP 31806)] [New Thread 0x7f0d4d283700 (LWP 31805)] [New Thread 0x7f0d4da84700 (LWP 31804)] [New Thread 0x7f0d4e285700 (LWP 31803)] [New Thread 0x7f0d4ea86700 (LWP 31802)] [New Thread 0x7f0d4f287700 (LWP 31801)] [New Thread 0x7f0d4fa88700 (LWP 31800)] [New Thread 0x7f0d50289700 (LWP 31799)] [New Thread 0x7f0d50a8a700 (LWP 31798)] [New Thread 0x7f0d5128b700 (LWP 31797)] [New Thread 0x7f0d51a8c700 (LWP 31796)] [New Thread 0x7f0d5228d700 (LWP 31795)] [New Thread 0x7f0d52a8e700 (LWP 31794)] [New Thread 0x7f0d5328f700 (LWP 31793)] [New Thread 0x7f0d53a90700 (LWP 31792)] [New Thread 0x7f0d54291700 (LWP 31791)] [New Thread 0x7f0d54a92700 (LWP 31790)] [New Thread 0x7f0d55293700 (LWP 31789)] [New Thread 0x7f0d55a94700 (LWP 31788)] [New Thread 0x7f0d56295700 (LWP 31787)] [New Thread 0x7f0d56a96700 (LWP 31786)] [New Thread 0x7f0d57297700 (LWP 31785)] [New Thread 0x7f0d57a98700 (LWP 31784)] [New Thread 0x7f0d58299700 (LWP 31783)] [New Thread 0x7f0d58a9a700 (LWP 31782)] [New Thread 0x7f0d6852d700 (LWP 31755)] [New Thread 0x7f0d68f31700 (LWP 31754)] Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0 Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2 Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6 Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6 Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1 Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6 Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done. Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /opt/couchbase/lib/libcouchstore.so...done. Loaded symbols for /opt/couchbase/lib/libcouchstore.so Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done. Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0 Reading symbols from /opt/couchbase/lib/libv8.so...done. Loaded symbols for /opt/couchbase/lib/libv8.so Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicui18n.so.44 Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicuuc.so.44 Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicudata.so.44 0x00007f0d693f72bd in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) t a a bt Thread 28 (Thread 0x7f0d68f31700 (LWP 31754)): #0 0x00007f0d6941cb73 in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f0d68f32c05 in get_command (buffer=0x7f0d68f30e50 "shutdown\n", buffsize=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/extensions/daemon/stdin_check.c:51 #2 check_stdin_thread (arg=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/extensions/daemon/stdin_check.c:93 #3 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2060) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 27 (Thread 0x7f0d6852d700 (LWP 31755)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=cond@entry=0x7f0d68730920, mutex=mutex@entry=0x7f0d687308e0, ms=19000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d6852fb7b in logger_thead_main (arg=0x2a12900) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/extensions/loggers/file_logger.c:342 #3 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2070) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 ---Type to continue, or q to quit--- #6 0x0000000000000000 in ?? () Thread 26 (Thread 0x7f0d58a9a700 (LWP 31782)): #0 0x00007f0d693f72bd in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f0d694219d4 in usleep () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f0d5b05a915 in updateStatsThread (arg=0x2a12040) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/memory_tracker.cc:36 #3 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2590) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 25 (Thread 0x7f0d58299700 (LWP 31783)): #0 0x00007f0d5b098795 in hasKey (k=..., this=0x14c85cdc0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/stored-value.h:145 #1 unlocked_find (bucket_num=, key=..., this=, wantsDeleted=, trackReference=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/stored-value.h:1199 #2 LoadValueCallback::callback (this=0x759d540, lookup=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.cc:347 #3 0x00007f0d5b0a7094 in CouchKVStore::recordDbDump (db=0x73fc880, docinfo=0x7624530, ctx=0x7f0d58298c20) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1564 ---Type to continue, or q to quit--- #4 0x00007f0d5adc2ae7 in lookup_callback (rq=, k=0x7f0d58298880, v=0x7f0d58298890) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:731 #5 0x00007f0d5adc11e8 in btree_lookup_inner (rq=rq@entry=0x7f0d58298b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:103 #6 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d58298b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #7 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d58298b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #8 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d58298b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #9 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=0x7f0d58298b80, diskpos=, current=0, end=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #10 0x00007f0d5adc2fff in couchstore_changes_since (db=0x73fc880, since=, options=, callback=, ctx=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:774 #11 0x00007f0d5b0a9b0f in CouchKVStore::loadDB (this=this@entry=0x748e000, cb=..., cl=..., sr=..., keysOnly=keysOnly@entry=false, vbid=185, startSeqno=startSeqno@entry=0, options=options@entry=4) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1223 #12 0x00007f0d5b0aa228 in CouchKVStore::dump (this=0x748e000, vbids=..., cb=..., cl=...) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1039 #13 0x00007f0d5b099886 in Warmup::loadDataforShard (this=0x2a219f0, shardId=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.cc:783 #14 0x00007f0d5b09c461 in WarmupLoadingData::run (this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.h:371 #15 0x00007f0d5b05c1af in ExecutorThread::run (this=this@entry=0x7376460) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:110 #16 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376460) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #17 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2f50) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #19 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #20 0x0000000000000000 in ?? () Thread 24 (Thread 0x7f0d57a98700 (LWP 31784)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372460, mutex=0x7372428, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372420, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 ---Type to continue, or q to quit--- #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372420, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372420, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=120 'x') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=120 'x') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7376540) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376540) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2f70) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 23 (Thread 0x7f0d57297700 (LWP 31785)): #0 0x00007f0d69471f10 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f0d6af75421 in snappy::SnappyDecompressor::RefillTag (this=this@entry=0x7f0d57296520) at snappy.cc:760 #2 0x00007f0d6af75def in DecompressAllTags (writer=, this=0x7f0d57296520) at snappy.cc:661 ---Type to continue, or q to quit--- #3 InternalUncompress (writer=, r=, max_len=) at snappy.cc:788 #4 snappy::RawUncompress (compressed=compressed@entry=0x7f0d572965a0, uncompressed=0x731c270 "ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336ﾭ\336", ...) at snappy.cc:956 #5 0x00007f0d6af7607c in snappy::RawUncompress (compressed=, n=, uncompressed=) at snappy.cc:951 #6 0x00007f0d5adc3bbd in pread_compressed (file=, pos=, ret_ptr=0x7f0d57296620) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_file_read.cc:177 #7 0x00007f0d5adc2db5 in bp_to_doc (options=, bp=, db=, pDoc=0x7f0d572966f0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:496 #8 couchstore_open_doc_with_docinfo (db=, docinfo=0x7b93b80, pDoc=0x7f0d572966f0, options=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:640 #9 0x00007f0d5b0a72a2 in CouchKVStore::recordDbDump (db=0x73fca00, docinfo=0x7b93b80, ctx=0x7f0d57296c20) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1589 #10 0x00007f0d5adc2ae7 in lookup_callback (rq=, k=0x7f0d57296880, v=0x7f0d57296890) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:731 #11 0x00007f0d5adc11e8 in btree_lookup_inner (rq=rq@entry=0x7f0d57296b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:103 #12 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d57296b80, diskpos=, ---Type to continue, or q to quit--- current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #13 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d57296b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #14 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d57296b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #15 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=0x7f0d57296b80, diskpos=, current=0, end=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #16 0x00007f0d5adc2fff in couchstore_changes_since (db=0x73fca00, since=, options=, callback=, ctx=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:774 #17 0x00007f0d5b0a9b0f in CouchKVStore::loadDB (this=this@entry=0x73b7400, cb=..., cl=..., sr=..., keysOnly=keysOnly@entry=false, vbid=660, startSeqno=startSeqno@entry=0, options=options@entry=4) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1223 #18 0x00007f0d5b0aa228 in CouchKVStore::dump (this=0x73b7400, vbids=..., cb=..., cl=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1039 #19 0x00007f0d5b099886 in Warmup::loadDataforShard (this=0x2a219f0, shardId=0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.cc:783 #20 0x00007f0d5b09c461 in WarmupLoadingData::run (this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.h:371 #21 0x00007f0d5b05c1af in ExecutorThread::run (this=this@entry=0x7376620) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:110 #22 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376620) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #23 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2f60) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #24 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #25 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #26 0x0000000000000000 in ?? () Thread 22 (Thread 0x7f0d56a96700 (LWP 31786)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372460, mutex=0x7372428, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372420, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372420, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372420, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=123 '{') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 ---Type to continue, or q to quit--- #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=123 '{') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7376700) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376700) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2f80) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 21 (Thread 0x7f0d56295700 (LWP 31787)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372460, mutex=0x7372428, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372420, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372420, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372420, thread=..., toSleep=true) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=136 '\210') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=136 '\210') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x73767e0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73767e0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2f90) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 20 (Thread 0x7f0d55a94700 (LWP 31788)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372460, mutex=0x7372428, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372420, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 ---Type to continue, or q to quit--- #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372420, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372420, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=182 '\266') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=182 '\266') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x73768c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73768c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2fa0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 19 (Thread 0x7f0d55293700 (LWP 31789)): #0 0x00007fff383fcf3c in gettimeofday () #1 0x00007f0d693e863a in gettimeofday () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f0d69c226ce in std::chrono::system_clock::now() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007f0d6b580d69 in gethrtime () ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cxxtime.cc:22 #4 0x00007f0d5b0b5571 in BlockTimer (o=0x0, n=0x0, d=, this=0x7f0d552924c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/histo.h:390 #5 cfs_pread (errinfo=0x73fc3a0, h=0x8082aa0, buf=0x8264431, sz=8192, off=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:103 #6 0x00007f0d5adc76ff in load_buffer_from (nbyte=1607, offset=187604992, buf=0x8264400, errinfo=0x73fc3a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/iobuffer.cc:167 #7 buffered_pread (errinfo=0x73fc3a0, handle=0x7e2e520, buf=0x7f0d55292580, nbyte=8, offset=187606591) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/iobuffer.cc:313 #8 0x00007f0d5adc3770 in read_skipping_prefixes (dst=0x7f0d55292580, len=, pos=, file=0x73fc380) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_file_read.cc:81 #9 pread_bin_internal (file=0x73fc380, pos=187606591, ret_ptr=ret_ptr@entry=0x7f0d552925d0, max_header_size=max_header_size@entry=0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_file_read.cc:113 #10 0x00007f0d5adc3b80 in pread_compressed (file=, pos=, ret_ptr=0x7f0d55292620) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_file_read.cc:159 #11 0x00007f0d5adc2db5 in bp_to_doc (options=, bp=, db=, pDoc=0x7f0d552926f0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:496 #12 couchstore_open_doc_with_docinfo (db=, docinfo=0x803f180, pDoc=0x7f0d552926f0, options=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:640 #13 0x00007f0d5b0a72a2 in CouchKVStore::recordDbDump (db=0x73fc380, docinfo=0x803f180, ctx=0x7f0d55292c20) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1589 ---Type to continue, or q to quit--- #14 0x00007f0d5adc2ae7 in lookup_callback (rq=, k=0x7f0d55292880, v=0x7f0d55292890) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:731 #15 0x00007f0d5adc11e8 in btree_lookup_inner (rq=rq@entry=0x7f0d55292b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:103 #16 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d55292b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #17 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d55292b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #18 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d55292b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #19 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=0x7f0d55292b80, diskpos=, current=0, end=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #20 0x00007f0d5adc2fff in couchstore_changes_since (db=0x73fc380, since=, options=, callback=, ctx=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:774 #21 0x00007f0d5b0a9b0f in CouchKVStore::loadDB (this=this@entry=0x7529400, cb=..., cl=..., sr=..., keysOnly=keysOnly@entry=false, vbid=127, startSeqno=startSeqno@entry=0, options=options@entry=4) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1223 #22 0x00007f0d5b0aa228 in CouchKVStore::dump (this=0x7529400, vbids=..., cb=..., cl=...) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1039 #23 0x00007f0d5b099886 in Warmup::loadDataforShard (this=0x2a219f0, shardId=3) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.cc:783 #24 0x00007f0d5b09c461 in WarmupLoadingData::run (this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.h:371 #25 0x00007f0d5b05c1af in ExecutorThread::run (this=this@entry=0x73769a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:110 #26 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73769a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #27 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2fb0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #28 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #29 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #30 0x0000000000000000 in ?? () Thread 18 (Thread 0x7f0d54a92700 (LWP 31790)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372460, mutex=0x7372428, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372420, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 ---Type to continue, or q to quit--- #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372420, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372420, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=135 '\207') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=135 '\207') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7376a80) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376a80) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2fc0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 17 (Thread 0x7f0d54291700 (LWP 31791)): #0 0x00007f0d6948212b in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f0d5b05f305 in hasKey (k=..., this=0x150606d00) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/stored-value.h:145 #2 unlocked_find (bucket_num=, key=..., this=, wantsDeleted=, ---Type to continue, or q to quit--- trackReference=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/stored-value.h:1199 #3 HashTable::insert (this=0x7985408, itm=..., policy=VALUE_ONLY, eject=false, partial=false) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/stored-value.cc:218 #4 0x00007f0d5b098c6b in LoadStorageKVPairCallback::callback (this=0x75842c0, val=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.cc:223 #5 0x00007f0d5b0a71da in CouchKVStore::recordDbDump (db=0x73ffe80, docinfo=0x803fa40, ctx=0x7f0d54290c20) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1633 #6 0x00007f0d5adc2ae7 in lookup_callback (rq=, k=0x7f0d54290880, v=0x7f0d54290890) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:731 #7 0x00007f0d5adc11e8 in btree_lookup_inner (rq=rq@entry=0x7f0d54290b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:103 #8 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d54290b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #9 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d54290b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #10 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=rq@entry=0x7f0d54290b80, diskpos=, current=current@entry=0, end=end@entry=1) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #11 0x00007f0d5adc0fd4 in btree_lookup_inner (rq=0x7f0d54290b80, diskpos=, current=0, end=1) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/btree_read.cc:71 #12 0x00007f0d5adc2fff in couchstore_changes_since (db=0x73ffe80, since=, options=, callback=, ctx=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/couchstore/src/couch_db.cc:774 #13 0x00007f0d5b0a9b0f in CouchKVStore::loadDB (this=this@entry=0x7490800, cb=..., cl=..., sr=..., keysOnly=keysOnly@entry=false, vbid=286, startSeqno=startSeqno@entry=0, options=options@entry=4) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1223 #14 0x00007f0d5b0aa228 in CouchKVStore::dump (this=0x7490800, vbids=..., cb=..., cl=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1039 #15 0x00007f0d5b099886 in Warmup::loadDataforShard (this=0x2a219f0, shardId=2) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.cc:783 #16 0x00007f0d5b09c461 in WarmupLoadingData::run (this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/warmup.h:371 #17 0x00007f0d5b05c1af in ExecutorThread::run (this=this@entry=0x7376b60) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:110 #18 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376b60) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #19 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2fd0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #20 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #21 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #22 0x0000000000000000 in ?? () ---Type to continue, or q to quit--- Thread 16 (Thread 0x7f0d53a90700 (LWP 31792)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372460, mutex=0x7372428, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372420, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372420, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372420, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=232 '\350') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=232 '\350') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7376c40) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376c40) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2fe0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 ---Type to continue, or q to quit--- #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 15 (Thread 0x7f0d5328f700 (LWP 31793)): #0 0x00007fff383fcf3c in gettimeofday () #1 0x00007f0d693e863a in gettimeofday () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f0d5b05bfe9 in ExecutorThread::run (this=this@entry=0x7376d20) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:90 #3 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376d20) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #4 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c2ff0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #5 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #6 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x0000000000000000 in ?? () Thread 14 (Thread 0x7f0d52a8e700 (LWP 31794)): #0 0x00007fff383fcf3c in gettimeofday () #1 0x00007f0d693e863a in gettimeofday () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f0d69c226ce in std::chrono::system_clock::now() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007f0d6b580d69 in gethrtime () at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cxxtime.cc:22 #4 0x00007f0d5b05c1c0 in ExecutorThread::run (this=this@entry=0x7376e00) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:113 #5 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376e00) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #6 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #7 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #8 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #9 0x0000000000000000 in ?? () Thread 13 (Thread 0x7f0d5228d700 (LWP 31795)): #0 0x00007f0d6a28fd5c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6a28b3a9 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f0d6a28b1cb in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007f0d6b580956 in cb_mutex_enter (mutex=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f0d5b05b54d in Mutex::acquire (this=0x73722c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f0d5b07cad1 in lock (this=0x7f0d5228cd00) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/locks.h:70 #6 LockHolder (m=..., this=0x7f0d5228cd00, tryLock=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/locks.h:47 #7 TaskQueue::_fetchNextTask (this=this@entry=0x73722c0, t=..., toSleep=toSleep@entry=false) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:96 ---Type to continue, or q to quit--- #8 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73722c0, thread=..., toSleep=false) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #9 0x00007f0d5b0486cb in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=154 '\232') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:209 #10 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=154 '\232') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #11 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7376ee0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #12 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376ee0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #13 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3010) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #14 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #16 0x0000000000000000 in ?? () Thread 12 (Thread 0x7f0d51a8c700 (LWP 31796)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372300, mutex=0x73722c8, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x73722c0, t=...) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x73722c0, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73722c0, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=211 '\323') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=211 '\323') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7376fc0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7376fc0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3020) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 11 (Thread 0x7f0d5128b700 (LWP 31797)): #0 0x00007f0d6a28fd5c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6a28b3a9 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f0d6a28b1cb in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 ---Type to continue, or q to quit--- #3 0x00007f0d6b580956 in cb_mutex_enter (mutex=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f0d5b05b54d in Mutex::acquire (this=0x73722c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f0d5b07b9f5 in lock (this=0x7f0d5128ad80) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/locks.h:70 #6 LockHolder (m=..., this=0x7f0d5128ad80, tryLock=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/locks.h:47 #7 _reschedule (curTaskType=@0x73770f8: NO_TASK_TYPE, task=..., this=0x73722c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:183 #8 TaskQueue::reschedule (this=0x73722c0, task=..., curTaskType=@0x73770f8: NO_TASK_TYPE) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:197 #9 0x00007f0d5b05c45c in ExecutorThread::run (this=this@entry=0x73770a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:136 #10 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73770a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #11 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3030) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #12 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #13 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #14 0x0000000000000000 in ?? () Thread 10 (Thread 0x7f0d50a8a700 (LWP 31798)): ---Type to continue, or q to quit--- #0 0x00007f0d6a28fd5c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6a28b3a9 in _L_lock_926 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f0d6a28b1cb in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0 #3 0x00007f0d6b580956 in cb_mutex_enter (mutex=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f0d5b05b54d in Mutex::acquire (this=0x73722c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f0d5b07cad1 in lock (this=0x7f0d50a89d00) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/locks.h:70 #6 LockHolder (m=..., this=0x7f0d50a89d00, tryLock=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/locks.h:47 #7 TaskQueue::_fetchNextTask (this=this@entry=0x73722c0, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:96 #8 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73722c0, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #9 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=112 'p') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #10 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=112 'p') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #11 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7377180) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #12 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7377180) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 ---Type to continue, or q to quit--- #13 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3040) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #14 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #16 0x0000000000000000 in ?? () Thread 9 (Thread 0x7f0d50289700 (LWP 31799)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372300, mutex=0x73722c8, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x73722c0, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x73722c0, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73722c0, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=41 ')') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=41 ')') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7377260) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7377260) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3050) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 8 (Thread 0x7f0d4fa88700 (LWP 31800)): #0 0x00007f0d5b095a49 in VBucketMap::getShard (this=0x73a0028, id=460) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/vbucketmap.cc:170 #1 0x00007f0d5b01b000 in EventuallyPersistentStore::flushVBucket (this=0x73a0000, vbid=460) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/ep.cc:2638 #2 0x00007f0d5b054628 in Flusher::flushVB (this=this@entry=0x2a30440) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/flusher.cc:283 #3 0x00007f0d5b054b78 in completeFlush (this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/flusher.cc:219 #4 Flusher::step (this=0x2a30440, task=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/flusher.cc:190 #5 0x00007f0d5b05c1af in ExecutorThread::run (this=this@entry=0x7377340) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:110 #6 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7377340) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #7 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3060) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #8 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #9 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #10 0x0000000000000000 in ?? () Thread 7 (Thread 0x7f0d4f287700 (LWP 31801)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x73725c0, mutex=0x7372588, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372580, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372580, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372580, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=230 '\346') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=230 '\346') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 ---Type to continue, or q to quit--- #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7377420) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7377420) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3070) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 6 (Thread 0x7f0d4ea86700 (LWP 31802)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x73725c0, mutex=0x7372588, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372580, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372580, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372580, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=230 '\346') ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=230 '\346') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7377500) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7377500) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3080) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 5 (Thread 0x7f0d4e285700 (LWP 31803)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x73725c0, mutex=0x7372588, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x7372580, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x7372580, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 ---Type to continue, or q to quit--- #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x7372580, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=230 '\346') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=230 '\346') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x73775e0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73775e0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c3090) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 4 (Thread 0x7f0d4da84700 (LWP 31804)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372720, mutex=0x73726e8, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x73726e0, t=...) ---Type to continue, or q to quit--- at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x73726e0, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73726e0, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=255 '\377') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=255 '\377') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x73776c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73776c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c30a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 3 (Thread 0x7f0d4d283700 (LWP 31805)): #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372720, mutex=0x73726e8, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 ---Type to continue, or q to quit--- #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x73726e0, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x73726e0, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73726e0, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=189 '\275') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=189 '\275') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x73777a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x73777a0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c30b0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #13 0x0000000000000000 in ?? () Thread 2 (Thread 0x7f0d4ca82700 (LWP 31806)): ---Type to continue, or q to quit--- #0 0x00007f0d6a28d6bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007f0d6b580af5 in cb_cond_timedwait (cond=0x7372720, mutex=0x73726e8, ms=2000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f0d5b078d61 in wait (tv=..., this=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/syncobject.h:74 #3 TaskQueue::_doSleep (this=this@entry=0x73726e0, t=...) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:77 #4 0x00007f0d5b07cbfb in TaskQueue::_fetchNextTask (this=this@entry=0x73726e0, t=..., toSleep=toSleep@entry=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:98 #5 0x00007f0d5b07cf1f in TaskQueue::fetchNextTask (this=0x73726e0, thread=..., toSleep=true) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:142 #6 0x00007f0d5b048775 in ExecutorPool::_nextTask (this=this@entry=0x7597840, t=..., tick=tick@entry=137 '\211') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:214 #7 0x00007f0d5b04882f in ExecutorPool::nextTask (this=0x7597840, t=..., tick=137 '\211') at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:229 #8 0x00007f0d5b05bfb3 in ExecutorThread::run (this=this@entry=0x7377880) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:78 #9 0x00007f0d5b05c769 in launch_executor_thread (arg=0x7377880) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f0d6b5807fa in platform_thread_wrap (arg=0x29c30c0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f0d6a288b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #12 0x00007f0d694277bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 ---Type to continue, or q to quit--- #13 0x0000000000000000 in ?? () Thread 1 (Thread 0x7f0d6bd9c720 (LWP 31694)): #0 0x00007f0d693f72bd in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f0d694219d4 in usleep () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007f0d5b052aea in Flusher::wait (this=0x2a30440) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/flusher.cc:47 #3 0x00007f0d5b0135bd in EventuallyPersistentStore::stopFlusher (this=this@entry=0x73a0000) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/ep.cc:402 #4 0x00007f0d5b018a27 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x73a0000, __in_chrg=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/ep.cc:364 #5 0x00007f0d5b032aed in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x7386000, __in_chrg=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5772 #6 0x00007f0d5b032ce5 in EvpDestroy (handle=0x7386000, force=false) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:143 #7 0x00007f0d67b21975 in bucket_shutdown_engine (key=, nkey=, val=0x7320c40, nval=, args=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1464 #8 0x00007f0d67b29e3c in genhash_iter (h=0x2a16000, iterfunc=iterfunc@entry=0x7f0d67b21940 , arg=arg@entry=0x0) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/engines/bucket_engine/genhash.c:262 ---Type to continue, or q to quit--- #9 0x00007f0d67b269be in bucket_destroy (handle=0x7f0d67d2c800, force=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1501 #10 bucket_destroy (handle=0x7f0d67d2c800, force=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1481 #11 0x0000000000406876 in main (argc=, argv=) at /home/buildbot/buildbot_slave/debian-7-x64-30x-builder/build/build/memcached/daemon/memcached.c:8853 (gdb) quit A debugging session is active. Inferior 1 [process 31694] will be detached. Quit anyway? (y or n) y Detaching from program: /opt/couchbase/bin/memcached, process 31694