GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /opt/couchbase/bin/memcached...done. Attaching to program: /opt/couchbase/bin/memcached, process 28282 Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done. Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1 Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done. Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done. Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done. Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done. Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5 Reading symbols from /lib/x86_64-linux-gnu/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libssl.so.0.9.8 Reading symbols from /lib/x86_64-linux-gnu/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libcrypto.so.0.9.8 Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libpthread-2.19.so...done. done. [New LWP 28585] [New LWP 28584] [New LWP 28583] [New LWP 28582] [New LWP 28581] [New LWP 28580] [New LWP 28579] [New LWP 28578] [New LWP 28577] [New LWP 28576] [New LWP 28575] [New LWP 28332] [New LWP 28331] [New LWP 28330] [New LWP 28329] [New LWP 28328] [New LWP 28327] [New LWP 28326] [New LWP 28325] [New LWP 28324] [New LWP 28323] [New LWP 28295] [New LWP 28294] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0 Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2 Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6 Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6 Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1 Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6 Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done. Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done. Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done. Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so Reading symbols from /opt/couchbase/lib/memcached/ep.so...done. Loaded symbols for /opt/couchbase/lib/memcached/ep.so Reading symbols from /opt/couchbase/lib/libcouchstore.so...done. Loaded symbols for /opt/couchbase/lib/libcouchstore.so Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done. Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0 Reading symbols from /opt/couchbase/lib/libv8.so...done. Loaded symbols for /opt/couchbase/lib/libv8.so Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicui18n.so.44 Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicuuc.so.44 Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done. Loaded symbols for /opt/couchbase/lib/libicudata.so.44 0x00007f93be05ab13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 Thread 24 (Thread 0x7f93bdb43700 (LWP 28294)): #0 0x00007f93be04d12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f93bdb44c45 in get_command (buffer=, buffsize=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/extensions/daemon/stdin_check.c:51 #2 check_stdin_thread (arg=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/extensions/daemon/stdin_check.c:93 #3 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0060) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f93bef59182 in start_thread (arg=0x7f93bdb43700) at pthread_create.c:312 #5 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (Thread 0x7f93bd13e700 (LWP 28295)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f93c01e777b in cb_cond_timedwait (cond=0x7f93bd342200 , mutex=0x7f93bd3421c0 , ms=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f93bd1413fa in logger_thead_main (arg=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/extensions/loggers/file_logger.c:405 #3 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0070) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f93bef59182 in start_thread (arg=0x7f93bd13e700) at pthread_create.c:312 #5 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (Thread 0x7f93bc730700 (LWP 28323)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x733a0d0) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x733a0d0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x733a0c8) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2a463 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder (tryLock=, m=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 EventuallyPersistentStore::setVBucketState (this=0x733a000, vbid=128, to=0, transfer=255, notify_dcp=208) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep.cc:1048 #8 0x00007f93b6ca4c9d in ActiveStream::setVBucketStateAckRecieved (this=0x108fdb180) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-stream.cc:457 #9 0x00007f93b6c9c1c9 in DcpProducer::handleResponse (this=, resp=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-producer.cc:497 #10 0x00007f93b6c389a9 in EvpDcpResponseHandler(ENGINE_HANDLE *, const void *, union {...} *) (handle=, cookie=0x7121e00, response=0x7140800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1688 #11 0x00007f93bc738c8d in dcp_response_handler (handle=, cookie=0x7121e00, response=0x7140800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:2636 #12 0x000000000040d939 in process_bin_dcp_response (c=0x7121e00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:8419 #13 0x000000000041948f in complete_nread (c=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5927 #14 conn_nread (c=0x733a0d0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7177 #15 0x000000000040cd3e in event_handler (fd=, which=, arg=0x7121e00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #16 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #17 event_process_active (base=) at event.c:1375 #18 event_base_loop (base=0x72ba280, flags=) at event.c:1572 #19 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b01a0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #20 0x00007f93bef59182 in start_thread (arg=0x7f93bc730700) at pthread_create.c:312 #21 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (Thread 0x7f93bbf2f700 (LWP 28324)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93bbf2ecc0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c81e67 in DcpConnMap::disconnect (this=0x732c000, cookie=0x71b7200) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1068 #8 0x00007f93b6c39f6b in EventuallyPersistentEngine::handleDisconnect (this=0x72d6a00, cookie=0x71b7200) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5708 #9 0x00007f93b6c3a093 in EvpHandleDisconnect(const void *, , const void *, const void *) (cookie=0x71b7200, type=, event_data=, cb_data=0x72d6a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1702 #10 0x00007f93bc7373b8 in handle_disconnect (cookie=0x71b7200, type=, event_data=, cb_data=0x7f93bc93d220 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1228 #11 0x000000000040d2b1 in perform_callbacks (c=, data=, type=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:267 #12 conn_immediate_close (c=0x71b7200) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7342 #13 0x000000000040cd3e in event_handler (fd=, which=, arg=0x71b7200) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #14 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #15 event_process_active (base=) at event.c:1375 #16 event_base_loop (base=0x72ba500, flags=) at event.c:1572 #17 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b01b0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f93bef59182 in start_thread (arg=0x7f93bbf2f700) at pthread_create.c:312 #19 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (Thread 0x7f93bb72e700 (LWP 28325)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93bb72dcc0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c81e67 in DcpConnMap::disconnect (this=0x732c000, cookie=0x71b7800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1068 #8 0x00007f93b6c39f6b in EventuallyPersistentEngine::handleDisconnect (this=0x72d6a00, cookie=0x71b7800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5708 #9 0x00007f93b6c3a093 in EvpHandleDisconnect(const void *, , const void *, const void *) (cookie=0x71b7800, type=, event_data=, cb_data=0x72d6a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1702 #10 0x00007f93bc7373b8 in handle_disconnect (cookie=0x71b7800, type=, event_data=, cb_data=0x7f93bc93d220 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1228 #11 0x000000000040d2b1 in perform_callbacks (c=, data=, type=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:267 #12 conn_immediate_close (c=0x71b7800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7342 #13 0x000000000040cd3e in event_handler (fd=, which=, arg=0x71b7800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #14 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #15 event_process_active (base=) at event.c:1375 #16 event_base_loop (base=0x72ba780, flags=) at event.c:1572 #17 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b01c0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f93bef59182 in start_thread (arg=0x7f93bb72e700) at pthread_create.c:312 #19 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (Thread 0x7f93baf2d700 (LWP 28326)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93baf2ccc0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c81e67 in DcpConnMap::disconnect (this=0x732c000, cookie=0x7247300) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1068 #8 0x00007f93b6c39f6b in EventuallyPersistentEngine::handleDisconnect (this=0x72d6a00, cookie=0x7247300) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5708 #9 0x00007f93b6c3a093 in EvpHandleDisconnect(const void *, , const void *, const void *) (cookie=0x7247300, type=, event_data=, cb_data=0x72d6a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1702 #10 0x00007f93bc7373b8 in handle_disconnect (cookie=0x7247300, type=, event_data=, cb_data=0x7f93bc93d220 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1228 #11 0x000000000040d2b1 in perform_callbacks (c=, data=, type=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:267 #12 conn_immediate_close (c=0x7247300) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7342 #13 0x000000000040cd3e in event_handler (fd=, which=, arg=0x7247300) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #14 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #15 event_process_active (base=) at event.c:1375 #16 event_base_loop (base=0x72baa00, flags=) at event.c:1572 #17 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b01d0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f93bef59182 in start_thread (arg=0x7f93baf2d700) at pthread_create.c:312 #19 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (Thread 0x7f93ba72c700 (LWP 28327)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93ba71b4f0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c816cf in ConnMap::findByName (this=0x732c000, name=...) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:206 #8 0x00007f93b6c41542 in EventuallyPersistentEngine::doDcpVbTakeoverStats (this=0x72d6a00, cookie=0x7247000, add_stat=, key=..., vbid=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5312 #9 0x00007f93b6c4e8e4 in EventuallyPersistentEngine::getStats (this=0x72d6a00, cookie=0x7247000, stat_key=0xd0e54018 "dcp-vbtakeover 335 replication:ns_1@172.23.105.62->ns_1@172.23.105.63:defaultt=0acca18ccd2ff468e4804d17a0f40115\",\"streamingUri\":\"/pools/default/bucketsStreaming/default?bucket_uuid=0acca18ccd2ff468e48"..., nkey=, add_stat=0x40b170 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:4465 #10 0x00007f93b6c4ebb6 in EvpGetStats (handle=0x72d6a00, cookie=0x7247000, stat_key=0xd0e54018 "dcp-vbtakeover 335 replication:ns_1@172.23.105.62->ns_1@172.23.105.63:defaultt=0acca18ccd2ff468e4804d17a0f40115\",\"streamingUri\":\"/pools/default/bucketsStreaming/default?bucket_uuid=0acca18ccd2ff468e48"..., nkey=77, add_stat=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:217 #11 0x00007f93bc734216 in bucket_get_stats (handle=, cookie=0x7247000, stat_key=0xd0e54018 "dcp-vbtakeover 335 replication:ns_1@172.23.105.62->ns_1@172.23.105.63:defaultt=0acca18ccd2ff468e4804d17a0f40115\",\"streamingUri\":\"/pools/default/bucketsStreaming/default?bucket_uuid=0acca18ccd2ff468e48"..., nkey=77, add_stat=0x40b170 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1908 #12 0x00000000004158b6 in stat_executor (c=0x7247000, packet=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5101 #13 0x000000000041931e in process_bin_packet (c=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5529 #14 complete_nread (c=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5935 #15 conn_nread (c=0x7247000) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7177 #16 0x000000000040cd3e in event_handler (fd=, which=, arg=0x7247000) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #17 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #18 event_process_active (base=) at event.c:1375 #19 event_base_loop (base=0x72bac80, flags=) at event.c:1572 #20 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b01e0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #21 0x00007f93bef59182 in start_thread (arg=0x7f93ba72c700) at pthread_create.c:312 #22 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (Thread 0x7f93b9f2b700 (LWP 28328)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93b9f2acc0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c81e67 in DcpConnMap::disconnect (this=0x732c000, cookie=0x7246d00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1068 #8 0x00007f93b6c39f6b in EventuallyPersistentEngine::handleDisconnect (this=0x72d6a00, cookie=0x7246d00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5708 #9 0x00007f93b6c3a093 in EvpHandleDisconnect(const void *, , const void *, const void *) (cookie=0x7246d00, type=, event_data=, cb_data=0x72d6a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1702 #10 0x00007f93bc7373b8 in handle_disconnect (cookie=0x7246d00, type=, event_data=, cb_data=0x7f93bc93d220 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1228 #11 0x000000000040d2b1 in perform_callbacks (c=, data=, type=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:267 #12 conn_immediate_close (c=0x7246d00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7342 #13 0x000000000040cd3e in event_handler (fd=, which=, arg=0x7246d00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #14 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #15 event_process_active (base=) at event.c:1375 #16 event_base_loop (base=0x72baf00, flags=) at event.c:1572 #17 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b01f0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f93bef59182 in start_thread (arg=0x7f93b9f2b700) at pthread_create.c:312 #19 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (Thread 0x7f93b972a700 (LWP 28329)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c4804a in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder (tryLock=, m=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 each (f=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.h:139 #8 EventuallyPersistentEngine::doConnAggStats (this=, cookie=0x7246a00, add_stat=0x40b170 , sepPtr=, sep_len=1, connType=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:3862 #9 0x00007f93b6c4e0a5 in EventuallyPersistentEngine::getStats (this=0x72d6a00, cookie=0x7246a00, stat_key=0x7272818 "dcpagg :details 959\"default\",\"uri\":\"/pools/default/buckets/default?bucket_uuid=0acca18ccd2ff468e4804d17a0f40115\",\"streamingUri\":\"/pools/default/bucketsStreaming/default?bucket_uuid=0acca18ccd2ff468e48"..., nkey=, add_stat=0x40b170 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:4363 #10 0x00007f93b6c4ebb6 in EvpGetStats (handle=0x72d6a00, cookie=0x7246a00, stat_key=0x7272818 "dcpagg :details 959\"default\",\"uri\":\"/pools/default/buckets/default?bucket_uuid=0acca18ccd2ff468e4804d17a0f40115\",\"streamingUri\":\"/pools/default/bucketsStreaming/default?bucket_uuid=0acca18ccd2ff468e48"..., nkey=8, add_stat=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:217 #11 0x00007f93bc734216 in bucket_get_stats (handle=, cookie=0x7246a00, stat_key=0x7272818 "dcpagg :details 959\"default\",\"uri\":\"/pools/default/buckets/default?bucket_uuid=0acca18ccd2ff468e4804d17a0f40115\",\"streamingUri\":\"/pools/default/bucketsStreaming/default?bucket_uuid=0acca18ccd2ff468e48"..., nkey=8, add_stat=0x40b170 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1908 #12 0x00000000004158b6 in stat_executor (c=0x7246a00, packet=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5101 #13 0x000000000041931e in process_bin_packet (c=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5529 #14 complete_nread (c=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:5935 #15 conn_nread (c=0x7246a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7177 #16 0x000000000040cd3e in event_handler (fd=, which=, arg=0x7246a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #17 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #18 event_process_active (base=) at event.c:1375 #19 event_base_loop (base=0x72bb180, flags=) at event.c:1572 #20 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0200) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #21 0x00007f93bef59182 in start_thread (arg=0x7f93b972a700) at pthread_create.c:312 #22 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (Thread 0x7f93b8f29700 (LWP 28330)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93b8f28cc0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c81e67 in DcpConnMap::disconnect (this=0x732c000, cookie=0x7246700) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1068 #8 0x00007f93b6c39f6b in EventuallyPersistentEngine::handleDisconnect (this=0x72d6a00, cookie=0x7246700) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5708 #9 0x00007f93b6c3a093 in EvpHandleDisconnect(const void *, , const void *, const void *) (cookie=0x7246700, type=, event_data=, cb_data=0x72d6a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1702 #10 0x00007f93bc7373b8 in handle_disconnect (cookie=0x7246700, type=, event_data=, cb_data=0x7f93bc93d220 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1228 #11 0x000000000040d2b1 in perform_callbacks (c=, data=, type=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:267 #12 conn_immediate_close (c=0x7246700) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7342 #13 0x000000000040cd3e in event_handler (fd=, which=, arg=0x7246700) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #14 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #15 event_process_active (base=) at event.c:1375 #16 event_base_loop (base=0x72bb400, flags=) at event.c:1572 #17 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0210) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f93bef59182 in start_thread (arg=0x7f93b8f29700) at pthread_create.c:312 #19 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (Thread 0x7f93b8728700 (LWP 28331)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93b8727cc0, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c81e67 in DcpConnMap::disconnect (this=0x732c000, cookie=0x71b7500) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1068 #8 0x00007f93b6c39f6b in EventuallyPersistentEngine::handleDisconnect (this=0x72d6a00, cookie=0x71b7500) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:5708 #9 0x00007f93b6c3a093 in EvpHandleDisconnect(const void *, , const void *, const void *) (cookie=0x71b7500, type=, event_data=, cb_data=0x72d6a00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:1702 #10 0x00007f93bc7373b8 in handle_disconnect (cookie=0x71b7500, type=, event_data=, cb_data=0x7f93bc93d220 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1228 #11 0x000000000040d2b1 in perform_callbacks (c=, data=, type=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:267 #12 conn_immediate_close (c=0x71b7500) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7342 #13 0x000000000040cd3e in event_handler (fd=, which=, arg=0x71b7500) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:7448 #14 0x00007f93bf767afc in event_process_active_single_queue (activeq=, base=) at event.c:1308 #15 event_process_active (base=) at event.c:1375 #16 event_base_loop (base=0x72bb680, flags=) at event.c:1572 #17 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0220) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #18 0x00007f93bef59182 in start_thread (arg=0x7f93b8728700) at pthread_create.c:312 #19 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (Thread 0x7f93b7f27700 (LWP 28332)): #0 0x00007f93be05ab13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f93bf77c5a6 in epoll_dispatch (base=0x72bb900, tv=) at epoll.c:404 #2 0x00007f93bf767a04 in event_base_loop (base=0x72bb900, flags=) at event.c:1558 #3 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0230) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f93bef59182 in start_thread (arg=0x7f93b7f27700) at pthread_create.c:312 #5 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (Thread 0x7f93b7726700 (LWP 28575)): #0 0x00007f93be020f3d in nanosleep () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f93be0524a4 in usleep (useconds=) at ../sysdeps/unix/sysv/linux/usleep.c:32 #2 0x00007f93b6c68965 in updateStatsThread (arg=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/memory_tracker.cc:36 #3 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b02d0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #4 0x00007f93bef59182 in start_thread (arg=0x7f93b7726700) at pthread_create.c:312 #5 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (Thread 0x7f93b4631700 (LWP 28576)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f93c01e777b in cb_cond_timedwait (cond=0x7308460, mutex=0x7308428, ms=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f93b6c8cfc0 in wait (nanoSecs=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/syncobject.h:49 #3 TaskQueue::_doSleep (this=0x7308420, t=...) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:94 #4 0x00007f93b6c8f66b in TaskQueue::_fetchNextTask (this=0x7308420, t=..., toSleep=true) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:117 #5 0x00007f93b6c8f86f in TaskQueue::fetchNextTask (this=0x7308420, thread=..., toSleep=136) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:161 #6 0x00007f93b6c57c89 in ExecutorPool::_nextTask (this=0x74a9a20, t=..., tick=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:217 #7 0x00007f93b6c57d0f in ExecutorPool::nextTask (this=0x74a9a20, t=..., tick=136 '\210') at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:232 #8 0x00007f93b6c6a221 in ExecutorThread::run (this=0x2a047e0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:81 #9 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x7308464) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0cb0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f93bef59182 in start_thread (arg=0x7f93b4631700) at pthread_create.c:312 #12 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (Thread 0x7f93b3e30700 (LWP 28577)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f93c01e777b in cb_cond_timedwait (cond=0x7308460, mutex=0x7308428, ms=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f93b6c8cfc0 in wait (nanoSecs=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/syncobject.h:49 #3 TaskQueue::_doSleep (this=0x7308420, t=...) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:94 #4 0x00007f93b6c8f66b in TaskQueue::_fetchNextTask (this=0x7308420, t=..., toSleep=true) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:117 #5 0x00007f93b6c8f86f in TaskQueue::fetchNextTask (this=0x7308420, thread=..., toSleep=134) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:161 #6 0x00007f93b6c57c89 in ExecutorPool::_nextTask (this=0x74a9a20, t=..., tick=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:217 #7 0x00007f93b6c57d0f in ExecutorPool::nextTask (this=0x74a9a20, t=..., tick=134 '\206') at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:232 #8 0x00007f93b6c6a221 in ExecutorThread::run (this=0x2a04900) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:81 #9 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x7308464) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0cd0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f93bef59182 in start_thread (arg=0x7f93b3e30700) at pthread_create.c:312 #12 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (Thread 0x7f93b362f700 (LWP 28578)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f93c01e777b in cb_cond_timedwait (cond=0x7308460, mutex=0x7308428, ms=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f93b6c8cfc0 in wait (nanoSecs=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/syncobject.h:49 #3 TaskQueue::_doSleep (this=0x7308420, t=...) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:94 #4 0x00007f93b6c8f66b in TaskQueue::_fetchNextTask (this=0x7308420, t=..., toSleep=true) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:117 #5 0x00007f93b6c8f86f in TaskQueue::fetchNextTask (this=0x7308420, thread=..., toSleep=132) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:161 #6 0x00007f93b6c57c89 in ExecutorPool::_nextTask (this=0x74a9a20, t=..., tick=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:217 #7 0x00007f93b6c57d0f in ExecutorPool::nextTask (this=0x74a9a20, t=..., tick=132 '\204') at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:232 #8 0x00007f93b6c6a221 in ExecutorThread::run (this=0x2a04a20) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:81 #9 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x7308464) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0cc0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f93bef59182 in start_thread (arg=0x7f93b362f700) at pthread_create.c:312 #12 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (Thread 0x7f93b2e2e700 (LWP 28579)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f93c01e777b in cb_cond_timedwait (cond=0x7308460, mutex=0x7308428, ms=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f93b6c8cfc0 in wait (nanoSecs=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/syncobject.h:49 #3 TaskQueue::_doSleep (this=0x7308420, t=...) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:94 #4 0x00007f93b6c8f66b in TaskQueue::_fetchNextTask (this=0x7308420, t=..., toSleep=true) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:117 #5 0x00007f93b6c8f86f in TaskQueue::fetchNextTask (this=0x7308420, thread=..., toSleep=138) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:161 #6 0x00007f93b6c57c89 in ExecutorPool::_nextTask (this=0x74a9a20, t=..., tick=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:217 #7 0x00007f93b6c57d0f in ExecutorPool::nextTask (this=0x74a9a20, t=..., tick=138 '\212') at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:232 #8 0x00007f93b6c6a221 in ExecutorThread::run (this=0x2a04b40) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:81 #9 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x7308464) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0ce0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f93bef59182 in start_thread (arg=0x7f93b2e2e700) at pthread_create.c:312 #12 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (Thread 0x7f93b262d700 (LWP 28580)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x32183020) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x32183020) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x32183018) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6ca7c11 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder (tryLock=, m=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 PassiveStream::addStats (this=0x32182ee0, add_stat=0x7f93b6c2be60 , c=0x7f93b262cc40) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-stream.cc:1533 #8 0x00007f93b6c94705 in DcpConsumer::addStats (this=0x82fc5a0, add_stat=, c=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-consumer.cc:564 #9 0x00007f93b6c3e1f9 in operator() (tc=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:3701 #10 for_each >, ConnStatBuilder> (__last=..., __first=..., __f=...) at /usr/include/c++/4.4/bits/stl_algo.h:4200 #11 each_UNLOCKED (f=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.h:148 #12 each (f=..., this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.h:140 #13 EventuallyPersistentEngine::doDcpStats (this=0x72d6a00, cookie=0x7f93b262cc40, add_stat=0x7f93b6c2be60 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:3954 #14 0x00007f93b6c4e1db in EventuallyPersistentEngine::getStats (this=0x72d6a00, cookie=0x7f93b262cc40, stat_key=0x7f93b6ce9104 "dcp", nkey=3, add_stat=0x7f93b6c2be60 ) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.cc:4367 #15 0x00007f93b6c2b922 in EventuallyPersistentStore::snapshotStats (this=0x733a000) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep.cc:1381 #16 0x00007f93b6c8ab14 in StatSnap::run (this=0x29e9ac0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/tasks.cc:80 #17 0x00007f93b6c6a5a5 in ExecutorThread::run (this=0x2a04c60) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:109 #18 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x32183020) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #19 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0cf0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #20 0x00007f93bef59182 in start_thread (arg=0x7f93b262d700) at pthread_create.c:312 #21 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (Thread 0x7f93b1e2c700 (LWP 28581)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x72d6468) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x72d6468) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x000000000041b316 in notify_io_complete (cookie=0x7122700, status=ENGINE_SUCCESS) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/thread.c:459 #5 0x00007f93b6c24b15 in notifyIOComplete (status=, cookie=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.h:489 #6 EventuallyPersistentStore::compactVBucket (this=, vbid=953, ctx=, cookie=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep.cc:1322 #7 0x00007f93b6c6a5a5 in ExecutorThread::run (this=0x2a04d80) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:109 #8 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x72d6468) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #9 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0d00) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #10 0x00007f93bef59182 in start_thread (arg=0x7f93b1e2c700) at pthread_create.c:312 #11 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (Thread 0x7f93b162b700 (LWP 28582)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x72d6648) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x72d6648) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x000000000041b316 in notify_io_complete (cookie=0x70d5b00, status=ENGINE_TMPFAIL) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/thread.c:459 #5 0x00007f93b6cabee1 in notifyIOComplete (status=, cookie=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.h:489 #6 VBucket::notifyOnPersistence (this=, e=..., idNum=, isBySeqno=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/vbucket.cc:362 #7 0x00007f93b6c254eb in EventuallyPersistentStore::flushVBucket (this=0x733a000, vbid=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep.cc:2850 #8 0x00007f93b6c60345 in Flusher::flushVB (this=0x2a0d900) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/flusher.cc:272 #9 0x00007f93b6c61c4d in Flusher::step (this=0x2a0d900, task=0x72da780) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/flusher.cc:173 #10 0x00007f93b6c6a5a5 in ExecutorThread::run (this=0x2a04ea0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:109 #11 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x72d6648) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #12 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0d10) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #13 0x00007f93bef59182 in start_thread (arg=0x7f93b162b700) at pthread_create.c:312 #14 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (Thread 0x7f93b0e2a700 (LWP 28583)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x72d6468) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x72d6468) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x000000000041b316 in notify_io_complete (cookie=0x70d5e00, status=ENGINE_TMPFAIL) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/thread.c:459 #5 0x00007f93b6cabee1 in notifyIOComplete (status=, cookie=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep_engine.h:489 #6 VBucket::notifyOnPersistence (this=, e=..., idNum=, isBySeqno=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/vbucket.cc:362 #7 0x00007f93b6c254eb in EventuallyPersistentStore::flushVBucket (this=0x733a000, vbid=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep.cc:2850 #8 0x00007f93b6c60345 in Flusher::flushVB (this=0x2a0d7c0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/flusher.cc:272 #9 0x00007f93b6c61c4d in Flusher::step (this=0x2a0d7c0, task=0x72da820) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/flusher.cc:173 #10 0x00007f93b6c6a5a5 in ExecutorThread::run (this=0x2a04fc0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:109 #11 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x72d6468) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #12 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0d20) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #13 0x00007f93bef59182 in start_thread (arg=0x7f93b0e2a700) at pthread_create.c:312 #14 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (Thread 0x7f93b0629700 (LWP 28584)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f93c01e777b in cb_cond_timedwait (cond=0x73085c0, mutex=0x7308588, ms=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:156 #2 0x00007f93b6c8cf3b in wait (secs=, this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/syncobject.h:44 #3 TaskQueue::_doSleep (this=0x7308580, t=...) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:92 #4 0x00007f93b6c8f66b in TaskQueue::_fetchNextTask (this=0x7308580, t=..., toSleep=true) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:117 #5 0x00007f93b6c8f86f in TaskQueue::fetchNextTask (this=0x7308580, thread=..., toSleep=33) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/taskqueue.cc:161 #6 0x00007f93b6c57c89 in ExecutorPool::_nextTask (this=0x74a9a20, t=..., tick=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:217 #7 0x00007f93b6c57d0f in ExecutorPool::nextTask (this=0x74a9a20, t=..., tick=33 '!') at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorpool.cc:232 #8 0x00007f93b6c6a221 in ExecutorThread::run (this=0x2a050e0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:81 #9 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x73085c4) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #10 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0d30) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #11 0x00007f93bef59182 in start_thread (arg=0x7f93b0629700) at pthread_create.c:312 #12 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (Thread 0x7f93afe28700 (LWP 28585)): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f93bef5b657 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0 #2 0x00007f93bef5b480 in __GI___pthread_mutex_lock (mutex=0x732c050) at ../nptl/pthread_mutex_lock.c:79 #3 0x00007f93c01e7529 in cb_mutex_enter (mutex=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:85 #4 0x00007f93b6c6967d in Mutex::acquire (this=0x732c048) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/mutex.cc:31 #5 0x00007f93b6c2f587 in lock (this=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:71 #6 LockHolder::LockHolder (this=0x7f93afe27b10, m=..., tryLock=false) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/locks.h:48 #7 0x00007f93b6c80ec7 in DcpConnMap::vbucketStateChanged (this=0x732c000, vbucket=866, state=vbucket_state_active) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/connmap.cc:1044 #8 0x00007f93b6c2a9de in EventuallyPersistentStore::setVBucketState (this=0x733a000, vbid=, to=vbucket_state_active, transfer=, notify_dcp=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/ep.cc:1057 #9 0x00007f93b6ca206d in PassiveStream::processSetVBucketState (this=0x32182ee0, state=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-stream.cc:1483 #10 0x00007f93b6ca6a33 in PassiveStream::processBufferedMessages (this=0x32182ee0, processed_bytes=@0x7f93afe27d1c: 0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-stream.cc:1313 #11 0x00007f93b6c938f1 in DcpConsumer::processBufferedItems (this=0x82fc5a0) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-consumer.cc:608 #12 0x00007f93b6c93a65 in Processer::run (this=0x82f5800) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/dcp-consumer.cc:48 #13 0x00007f93b6c6a5a5 in ExecutorThread::run (this=0x2a05200) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:109 #14 0x00007f93b6c6aa8d in launch_executor_thread (arg=0x732c050) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/ep-engine/src/executorthread.cc:34 #15 0x00007f93c01e731a in platform_thread_wrap (arg=0x29b0d40) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/platform/src/cb_pthreads.c:19 #16 0x00007f93bef59182 in start_thread (arg=0x7f93afe28700) at pthread_create.c:312 #17 0x00007f93be05a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 1 (Thread 0x7f93c0a06780 (LWP 28282)): #0 0x00007f93be05ab13 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f93bf77c5a6 in epoll_dispatch (base=0x72ba000, tv=) at epoll.c:404 #2 0x00007f93bf767a04 in event_base_loop (base=0x72ba000, flags=) at event.c:1558 #3 0x000000000041177d in main (argc=, argv=) at /home/buildbot/buildbot_slave/ubuntu-1004-x64-30x-builder/build/build/memcached/daemon/memcached.c:8931 (gdb) quit A debugging session is active. Inferior 1 [process 28282] will be detached. Quit anyway? (y or n) [answered Y; input not from terminal] Detaching from program: /opt/couchbase/bin/memcached, process 28282