GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-119.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: . Attaching to process 45886 Reading symbols from /opt/couchbase/bin/memcached...Reading symbols from /usr/lib/debug/opt/couchbase/bin/memcached.debug...done. done. Reading symbols from /opt/couchbase/bin/../lib/libnuma.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libnuma.so.1.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libnuma.so.1 Reading symbols from /opt/couchbase/bin/../lib/crash_engine.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/crash_engine.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/crash_engine.so Reading symbols from /opt/couchbase/bin/../lib/libdefault_engine.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libdefault_engine.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libdefault_engine.so Reading symbols from /opt/couchbase/bin/../lib/libep.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libep.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libep.so Reading symbols from /opt/couchbase/bin/../lib/libstatistics.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libstatistics.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libstatistics.so Reading symbols from /opt/couchbase/bin/../lib/libcouchstore.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcouchstore.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libcouchstore.so Reading symbols from /opt/couchbase/bin/../lib/libmagma.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libmagma.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libmagma.so Reading symbols from /opt/couchbase/bin/../lib/libmemcached_logger.so.1.0.0...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libmemcached_logger.so.1.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libmemcached_logger.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libengine_utilities.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libengine_utilities.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libengine_utilities.so Reading symbols from /opt/couchbase/bin/../lib/libz.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libz.so.1.2.11.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libz.so.1 Reading symbols from /opt/couchbase/bin/../lib/libcbcompress.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcbcompress.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libcbcompress.so Reading symbols from /opt/couchbase/bin/../lib/liblz4.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/liblz4.so.1.9.2.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/liblz4.so.1 Reading symbols from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libplatform_so.so.0.1.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 Reading symbols from /opt/couchbase/bin/../lib/libphosphor.so...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libphosphor.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libphosphor.so Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libJSON_checker.so.1.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.2.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libstdc++.so.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [New LWP 46394] [New LWP 46393] [New LWP 46392] [New LWP 46391] [New LWP 46390] [New LWP 46389] [New LWP 46388] [New LWP 46387] [New LWP 46386] [New LWP 46385] [New LWP 46384] [New LWP 46383] [New LWP 46382] [New LWP 46381] [New LWP 46380] [New LWP 46379] [New LWP 46378] [New LWP 46377] [New LWP 46376] [New LWP 46375] [New LWP 46374] [New LWP 46373] [New LWP 46372] [New LWP 46371] [New LWP 46370] [New LWP 46369] [New LWP 46368] [New LWP 46367] [New LWP 46366] [New LWP 46365] [New LWP 46364] [New LWP 46363] [New LWP 46362] [New LWP 46361] [New LWP 46360] [New LWP 46359] [New LWP 46358] [New LWP 46357] [New LWP 46356] [New LWP 45909] [New LWP 45908] [New LWP 45907] [New LWP 45906] [New LWP 45905] [New LWP 45904] [New LWP 45903] [New LWP 45902] [New LWP 45901] [New LWP 45900] [New LWP 45899] [New LWP 45898] [New LWP 45897] [New LWP 45896] [New LWP 45895] [New LWP 45894] [New LWP 45893] [New LWP 45892] [New LWP 45891] [New LWP 45890] [New LWP 45889] [New LWP 45888] [New LWP 45887] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 Thread 63 (Thread 0x7fdc1f3ff700 (LWP 45887)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc21d598c9 in background_thread_sleep (tsdn=, interval=, info=) at src/background_thread.c:232 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7fdc1f3e6a50) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7fdc1f3e6a50) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 62 (Thread 0x7fdc1e67e700 (LWP 45888)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc235838bd in __gthread_cond_timedwait (__abs_timeout=0x7fdc1e664fa0, __mutex=, __cond=0x7fdc1f82e138) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fdc1f82e138) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fdc1f82e138) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __atime=..., __lock=..., this=0x7fdc1f82e138) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7fdc1f82e138) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7fdc1f82e110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/mpmc_blocking_q.h:61 #7 spdlog::details::thread_pool::process_next_msg_ (this=this@entry=0x7fdc1f82e110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:193 #8 0x00007fdc23583e10 in spdlog::details::thread_pool::worker_loop_ (this=0x7fdc1f82e110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:184 #9 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1f858460) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #10 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 61 (Thread 0x7fdc1de7d700 (LWP 45889)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc23582f7a in __gthread_cond_timedwait (__abs_timeout=0x7fdc1de641f0, __mutex=, __cond=) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7fdc1f9ffa88) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/periodic_worker.h:38 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > ( __fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7fdc1f9ffa88) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7fdc1f9ffa88) at /usr/local/include/c++/7.3.0/thread:243 #11 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv (this=0x7fdc1f9ffa80) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1f9ffa80) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7fdc1d67c700 (LWP 45890)): #0 0x00007fdc1ffa3c3d in poll () from /lib64/libc.so.6 #1 0x000000000046adde in get_command (buffsize=80, buffer=0x7fdc1d6631d0 " F\243\"\334\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:49 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:90 #3 0x00007fdc22a4d727 in run (this=0x7fdc1f9f63f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #4 platform_thread_wrap (arg=0x7fdc1f9f63f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #5 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7fdc1ce7b700 (LWP 45891)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc24178c62 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=0x7fdb90471df8, state=@0x7fdc1ce5d78c: 385, token=0x7fdc1ce5d7b8, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/detail/Futex-inl.h:101 #4 0x00007fdc24178ca8 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=, token=, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:1298 #5 0x00007fdc242ca508 in ReadHolder (lock=..., this=0x7fdc1ce5d7b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:419 #6 VBucket::getSyncWriteAcceptedCount (this=0x7fdb90471c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:306 #7 0x00007fdc242c2f5a in visitBucket (vb=..., this=0x7fdb5fc51bc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:80 #8 VBucketCountAggregator::visitBucket (this=, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:90 #9 0x00007fdc24284d85 in KVBucket::visit (this=0x7fdc1c1a0000, visitor=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2225 #10 0x00007fdc24286b53 in KVBucket::getAggregatedVBucketStats (this=0x7fdc1c1a0000, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1189 #11 0x00007fdc242247ce in EventuallyPersistentEngine::doEngineStats (this=0x7fdc1c0fb000, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2941 #12 0x00007fdc24238196 in EventuallyPersistentEngine::get_prometheus_stats (this=0x7fdc1c0fb000, collector=..., cardinality=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:389 #13 0x000000000046ac38 in operator() (bucket=..., __closure=0x7fdc1ce5db70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stats.cc:175 #14 std::_Function_handler >::_M_invoke(const std::_Any_data &, Bucket &, ) ( __functor=..., __args#0=..., __args#1=) at /usr/local/include/c++/7.3.0/bits/std_function.h:302 #15 0x0000000000420db1 in bucketsForEach(std::function, void*) (fn=..., arg=arg@entry=0x0) at /usr/local/include/c++/7.3.0/bits/std_function.h:706 #16 0x00000000004684c9 in server_prometheus_stats (collector=..., cardinality=cb::prometheus::Low) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stats.cc:166 #17 0x00007fdc23e1cd32 in operator() (__args#1=, __args#0=..., this=0x7fdc1e687020) at /usr/local/include/c++/7.3.0/bits/std_function.h:706 #18 cb::prometheus::MetricServer::KVCollectable::Collect (this=0x7fdc1e687010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/prometheus.cc:106 #19 0x00007fdc23e3c82b in prometheus::detail::CollectMetrics (collectables=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/pull/src/metrics_collector.cc:18 #20 0x00007fdc23e3b6a8 in prometheus::detail::MetricsHandler::handleGet (this=0x7fdc1f82d700, conn=0xcc5950) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/pull/src/handler.cc:129 #21 0x00007fdc23e29737 in CivetServer::requestHandler (conn=0xcc5950, cbdata=0x7fdc1f82d700) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/CivetServer.cpp:152 #22 0x00007fdc23e36339 in handle_request (conn=conn@entry=0xcc5950) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:14566 #23 0x00007fdc23e37649 in process_new_connection (conn=0xcc5950) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18545 #24 worker_thread_run (conn=0xcc5950) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18947 #25 worker_thread (thread_func_param=0xcc5950) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19002 #26 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #27 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7fdc17fff700 (LWP 45892)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000468491 in __gthread_mutex_lock (__mutex=) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 lock (this=) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 #5 lock_guard (__m=..., this=) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:162 #6 server_prometheus_stats (collector=..., cardinality=cb::prometheus::High) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stats.cc:159 #7 0x00007fdc23e1cd32 in operator() (__args#1=, __args#0=..., this=0x7fdc1f9ff9a0) at /usr/local/include/c++/7.3.0/bits/std_function.h:706 #8 cb::prometheus::MetricServer::KVCollectable::Collect (this=0x7fdc1f9ff990) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/prometheus.cc:106 #9 0x00007fdc23e3c82b in prometheus::detail::CollectMetrics (collectables=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/pull/src/metrics_collector.cc:18 #10 0x00007fdc23e3b6a8 in prometheus::detail::MetricsHandler::handleGet (this=0x7fdc1f82d680, conn=0xcc6338) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/pull/src/handler.cc:129 #11 0x00007fdc23e29737 in CivetServer::requestHandler (conn=0xcc6338, cbdata=0x7fdc1f82d680) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/CivetServer.cpp:152 #12 0x00007fdc23e36339 in handle_request (conn=conn@entry=0xcc6338) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:14566 #13 0x00007fdc23e37649 in process_new_connection (conn=0xcc6338) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18545 #14 worker_thread_run (conn=0xcc6338) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18947 #15 worker_thread (thread_func_param=0xcc6338) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19002 #16 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7fdc177fe700 (LWP 45893)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc23e2df0b in produce_socket (sp=0x7fdc177e5140, ctx=0xcc4d20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18762 #2 accept_new_connection (ctx=0xcc4d20, listener=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19089 #3 master_thread_run (ctx=0xcc4d20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19157 #4 master_thread (thread_func_param=0xcc4d20) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19243 #5 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7fdc16ffd700 (LWP 45894)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000496e60 in __gthread_cond_timedwait (__abs_timeout=0x7fdc16fe41d0, __mutex=, __cond=0x7fdc1f82f0f8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=, this=0x7fdc1f82f0f8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=, this=0x7fdc1f82f0f8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=, __lock=, this=0x7fdc1f82f0f8) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 ExternalAuthManagerThread::run (this=0x7fdc1f82f000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/external_auth_manager_thread.cc:229 #6 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1f82f000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #7 0x00007fdc22a4d727 in run (this=0x7fdc1f9f6b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #8 platform_thread_wrap (arg=0x7fdc1f9f6b40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #9 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7fdc167fc700 (LWP 45895)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000004f5de6 in __gthread_cond_timedwait (__abs_timeout=0x7fdc167e31f0, __mutex=, __cond=0x7fdc1e68d8f8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fdc1e68d8f8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fdc1e68d8f8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=..., __lock=..., this=0x7fdc1e68d8f8) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 AuditImpl::consume_events (this=0x7fdc1e68d500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/audit.cc:408 #6 0x00007fdc22a4d727 in run (this=0x7fdc1f9f68a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #7 platform_thread_wrap (arg=0x7fdc1f9f68a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #8 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7fdc15ffb700 (LWP 45896)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000421021 in disassociate_bucket (connection=...) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 0x000000000048a6b6 in Connection::executeCommandsCallback (this=0x7fdc1c5b8000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:744 #5 0x000000000048ac15 in Connection::event_callback (event=, ctx=0x7fdc1c5b8000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:913 #6 0x00007fdc21b0de70 in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7fdc1c56e780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:225 #7 0x00007fdc21b16d01 in event_process_active_single_queue (base=base@entry=0x7fdc1f81f300, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #8 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f81f300) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #9 event_base_loop (base=0x7fdc1f81f300, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #10 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94b000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #11 0x00007fdc22a4d727 in run (this=0x7fdc1f83ae00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7fdc1f83ae00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7fdc157fa700 (LWP 45897)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000421021 in disassociate_bucket (connection=...) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 0x000000000048a6b6 in Connection::executeCommandsCallback (this=0x7fdc1c310000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:744 #5 0x000000000048ac15 in Connection::event_callback (event=, ctx=0x7fdc1c310000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:913 #6 0x00007fdc21b0de70 in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7fdc1f82fa00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:225 #7 0x00007fdc21b16d01 in event_process_active_single_queue (base=base@entry=0x7fdc1f81f600, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #8 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f81f600) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #9 event_base_loop (base=0x7fdc1f81f600, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #10 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94baa8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #11 0x00007fdc22a4d727 in run (this=0x7fdc1f83afe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7fdc1f83afe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7fdc14ff9700 (LWP 45898)): #0 0x00007fdc1ffaeeb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fdc21b203c5 in epoll_dispatch (base=0x7fdc1f81f900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fdc21b17376 in event_base_loop (base=0x7fdc1f81f900, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94c550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #4 0x00007fdc22a4d727 in run (this=0x7fdc1f83b0d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #5 platform_thread_wrap (arg=0x7fdc1f83b0d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #6 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7fdc0ffff700 (LWP 45899)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000421021 in disassociate_bucket (connection=...) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 0x000000000048a6b6 in Connection::executeCommandsCallback (this=0x7fdc1c38d000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:744 #5 0x000000000048ac15 in Connection::event_callback (event=, ctx=0x7fdc1c38d000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:913 #6 0x00007fdc21b0de70 in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7fdc1c364780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:225 #7 0x00007fdc21b16d01 in event_process_active_single_queue (base=base@entry=0x7fdc1f81fc00, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #8 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f81fc00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #9 event_base_loop (base=0x7fdc1f81fc00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #10 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94cff8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #11 0x00007fdc22a4d727 in run (this=0x7fdc1f83af80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7fdc1f83af80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7fdc0f7fe700 (LWP 45900)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000421021 in disassociate_bucket (connection=...) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 0x000000000048a6b6 in Connection::executeCommandsCallback (this=0x7fdc1c37b000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:744 #5 0x000000000048ac15 in Connection::event_callback (event=, ctx=0x7fdc1c37b000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:913 #6 0x00007fdc21b0de70 in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7fdc1c3d3780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:225 #7 0x00007fdc21b16d01 in event_process_active_single_queue (base=base@entry=0x7fdc1f81ff00, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #8 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f81ff00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #9 event_base_loop (base=0x7fdc1f81ff00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #10 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94daa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #11 0x00007fdc22a4d727 in run (this=0x7fdc1f83b130) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7fdc1f83b130) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7fdc0effd700 (LWP 45901)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000421021 in disassociate_bucket (connection=...) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 0x000000000048a6b6 in Connection::executeCommandsCallback (this=0x7fdc1c3e3000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:744 #5 0x000000000048ac15 in Connection::event_callback (event=, ctx=0x7fdc1c3e3000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:913 #6 0x00007fdc21b0de70 in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7fdc1c403780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:225 #7 0x00007fdc21b16d01 in event_process_active_single_queue (base=base@entry=0x7fdc1f820200, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #8 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f820200) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #9 event_base_loop (base=0x7fdc1f820200, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #10 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94e548) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #11 0x00007fdc22a4d727 in run (this=0x7fdc1f83af20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7fdc1f83af20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7fdc0e7fc700 (LWP 45902)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc24178c62 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=0x7fdb90471df8, state=@0x7fdc0e7e239c: 385, token=0x7fdc0e7e23c8, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/detail/Futex-inl.h:101 #4 0x00007fdc24178ca8 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=, token=, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:1298 #5 0x00007fdc242ca508 in ReadHolder (lock=..., this=0x7fdc0e7e23c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:419 #6 VBucket::getSyncWriteAcceptedCount (this=0x7fdb90471c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:306 #7 0x00007fdc242c2f5a in visitBucket (vb=..., this=0x7fdb90456800) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:80 #8 VBucketCountAggregator::visitBucket (this=, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:90 #9 0x00007fdc24284d85 in KVBucket::visit (this=0x7fdc1c1a0000, visitor=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2225 #10 0x00007fdc24286b53 in KVBucket::getAggregatedVBucketStats (this=0x7fdc1c1a0000, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1189 #11 0x00007fdc242247ce in EventuallyPersistentEngine::doEngineStats (this=this@entry=0x7fdc1c0fb000, collector=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2941 #12 0x00007fdc24238c1b in EventuallyPersistentEngine::getStats(void const*, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, gsl::not_null)> const&) (this=this@entry=0x7fdc1c0fb000, cookie=0x7fdc1c557a00, key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:4660 #13 0x00007fdc24239a97 in EventuallyPersistentEngine::get_stats(gsl::not_null, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, gsl::not_null)> const&) (this=0x7fdc1c0fb000, cookie=..., key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:371 #14 0x000000000044ee82 in bucket_get_stats(Cookie&, std::basic_string_view >, cb::sized_buffer, std::function >, std::basic_string_view >, gsl::not_null)> const&) (cookie=..., key=..., value=..., add_stat=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:381 #15 0x00000000004d9fca in stat_all_stats (arg=..., cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:490 #16 0x00000000004d9c4d in StatsCommandContext::doStats (this=this@entry=0x7fdc1e693ce0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:727 #17 0x00000000004dad38 in StatsCommandContext::step (this=0x7fdc1e693ce0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:649 #18 0x00000000004ddbb0 in SteppableCommandContext::drive (this=this@entry=0x7fdc1e693ce0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:42 #19 0x00000000004b19d6 in stat_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:192 #20 0x00000000004b48f6 in operator() (__args#0=..., this=) at /usr/local/include/c++/7.3.0/bits/std_function.h:706 #21 execute_client_request_packet (cookie=..., request=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:920 #22 0x00000000004a2669 in Cookie::execute (this=0x7fdc1c557a00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:158 #23 0x000000000048353d in Connection::executeCommandPipeline (this=this@entry=0x7fdc1c533000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:553 #24 0x000000000048a4d5 in Connection::executeCommandsCallback (this=this@entry=0x7fdc1c533000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:679 #25 0x000000000048b39f in Connection::rw_callback (ctx=0x7fdc1c533000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:808 #26 0x00007fdc21b0dd8e in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7fdc1c557780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208 #27 0x00007fdc21b16d01 in event_process_active_single_queue (base=base@entry=0x7fdc1f820500, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #28 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f820500) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #29 event_base_loop (base=0x7fdc1f820500, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #30 0x000000000046bb69 in worker_libevent (arg=0x7fdc1e94eff0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:195 #31 0x00007fdc22a4d727 in run (this=0x7fdc1f83af50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #32 platform_thread_wrap (arg=0x7fdc1f83af50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #33 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #34 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7fdc0dffb700 (LWP 45903)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47c158) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47c158, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47c000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47c000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7fdc0d7fa700 (LWP 45904)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47c318) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47c318, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47c1c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47c1c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b1f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b1f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7fdc0cff9700 (LWP 45905)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47c4d8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47c4d8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47c380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47c380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b0a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b0a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fdc07fff700 (LWP 45906)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47c698) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47c698, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47c540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47c540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b250) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b250) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fdc077fe700 (LWP 45907)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47c858) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47c858, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47c700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47c700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fdc06ffd700 (LWP 45908)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47ca18) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47ca18, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47c8c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47c8c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b070) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b070) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fdc067fc700 (LWP 45909)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c47cbd8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c47cbd8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000004a7921 in Executor::run (this=0x7fdc1c47ca80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #4 0x00007fdc22a63c8f in Couchbase::Thread::thread_entry (this=0x7fdc1c47ca80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #5 0x00007fdc22a4d727 in run (this=0x7fdc1f83b220) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1f83b220) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fdbc7fff700 (LWP 46356)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc21d58fa2 in background_thread_sleep (tsdn=, interval=, info=) at src/background_thread.c:232 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fdbc7fe6a50) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fdbc77fe700 (LWP 46357)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27dc920) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27dc920, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27dc920, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1ebf9f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1ebf9f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c3dc9d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c3dc9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2ca0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fdbc6ffd700 (LWP 46358)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27dcd70) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27dcd70, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27dcd70, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e6080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c3dca20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c3dca10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2d10) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fdbc67fc700 (LWP 46359)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27dd910) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27dd910, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27dd910, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e6380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c3dcb10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c3dcb00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2df0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fdbc5ffb700 (LWP 46360)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27dcf68) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27dcf68, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27dcf68, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e6680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c3fd510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c3fd500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2e60) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fdbc57fa700 (LWP 46361)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27dfdb8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27dfdb8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27dfdb8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e66c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e66c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c400b20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c400b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2ed0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fdbc4ff9700 (LWP 46362)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27dd8e0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27dd8e0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27dd8e0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e6440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c400f30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c400f20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2f40) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fdbb27da700 (LWP 46363)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27e0100) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27e0100, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27e0100, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e6700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c4016b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c4016a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d2fb0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fdbb1fd9700 (LWP 46364)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27e00a0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27e00a0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27e00a0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc0000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc0000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14060800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6740) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1c3e6740) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c40c350) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c40c340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3020) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fdbb17d8700 (LWP 46365)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fdc2429019c in __gthread_mutex_lock (__mutex=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 lock (this=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 #5 lock (this=0x7fdbb17beac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:267 #6 unique_lock (__m=..., this=0x7fdbb17beac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:197 #7 KVBucket::rollback (this=0x7fdc1c1a0000, vbid=..., rollbackSeqno=4882) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2414 #8 0x00007fdc241b5fab in DcpConsumer::doRollback (this=0x7fdb905e9040, opaque=58, vbid=..., rollbackSeqno=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1126 #9 0x00007fdc241b6140 in RollbackTask::run (this=0x7fd981d67a50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:966 #10 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd981d67a50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #11 0x00007fdc24263782 in operator() (__closure=0x7fdbb17bf040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #12 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #13 0x00007fdc243da470 in operator() (this=0x7fdbb17bf040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #14 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc14061400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #15 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14061400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #16 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #17 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #18 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #19 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #20 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #21 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fdc1c3e6840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fdc24261c73 in operator() (this=0x7fdc1c40d200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #25 operator() (__closure=0x7fdc1c40d1f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3090) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #28 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fdbb0fd7700 (LWP 46366)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fdc2429019c in __gthread_mutex_lock (__mutex=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 lock (this=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 #5 lock (this=0x7fdbb0fbdac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:267 #6 unique_lock (__m=..., this=0x7fdbb0fbdac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:197 #7 KVBucket::rollback (this=0x7fdc1c1a0000, vbid=..., rollbackSeqno=4880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2414 #8 0x00007fdc241b5fab in DcpConsumer::doRollback (this=0x7fdb905e9040, opaque=55, vbid=..., rollbackSeqno=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1126 #9 0x00007fdc241b6140 in RollbackTask::run (this=0x7fd981d67810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:966 #10 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd981d67810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #11 0x00007fdc24263782 in operator() (__closure=0x7fdbb0fbe040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #12 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #13 0x00007fdc243da470 in operator() (this=0x7fdbb0fbe040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #14 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc14061400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #15 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14061400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #16 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #17 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #18 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #19 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #20 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #21 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e68c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fdc1c3e68c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fdc24261c73 in operator() (this=0x7fdc1c40d250) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #25 operator() (__closure=0x7fdc1c40d240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3100) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #28 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fdb93fff700 (LWP 46367)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fdc2429019c in __gthread_mutex_lock (__mutex=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 lock (this=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 #5 lock (this=0x7fdb93fe5ac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:267 #6 unique_lock (__m=..., this=0x7fdb93fe5ac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:197 #7 KVBucket::rollback (this=0x7fdc1c1a0000, vbid=..., rollbackSeqno=4862) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2414 #8 0x00007fdc241b5fab in DcpConsumer::doRollback (this=0x7fdb905e9040, opaque=52, vbid=..., rollbackSeqno=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1126 #9 0x00007fdc241b6140 in RollbackTask::run (this=0x7fd981d675d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:966 #10 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd981d675d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #11 0x00007fdc24263782 in operator() (__closure=0x7fdb93fe6040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #12 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #13 0x00007fdc243da470 in operator() (this=0x7fdb93fe6040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #14 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc14061400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #15 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14061400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #16 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #17 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #18 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #19 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #20 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #21 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6940) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fdc1c3e6940) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fdc24261c73 in operator() (this=0x7fdc1c40d2a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #25 operator() (__closure=0x7fdc1c40d290) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3170) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #28 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fdb937fe700 (LWP 46368)): #0 load (__m=std::memory_order_seq_cst, this=0x7fdb5d1f3950) at /usr/local/include/c++/7.3.0/bits/atomic_base.h:396 #1 operator std::__atomic_base::__int_type (this=0x7fdb5d1f3950) at /usr/local/include/c++/7.3.0/bits/atomic_base.h:259 #2 magma::LSMTree::resetTree (this=0x7fdb5d1f3810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:238 #3 0x00007fdc23886205 in magma::RollbackTransaction::Finalize (this=0x7fdbc40c2760) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/rollback.cc:60 #4 0x00007fdc2387781f in magma::KVStore::rollback(magma::Magma::WriteOperation const&, magma::WAL*, std::function, magma::WALOffset) (this=this@entry=0x7fdb5d0fda10, op=..., wal=wal@entry=0x7fdc1c245180, callback=..., rbWALOffset=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:902 #5 0x00007fdc23877e41 in magma::KVStore::Rollback(magma::Magma::WriteOperation const&, magma::WAL*, std::function) (this=this@entry=0x7fdb5d0fda10, op=..., wal=0x7fdc1c245180, callback=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:751 #6 0x00007fdc2383d192 in magma::Magma::Impl::Rollback(unsigned short, unsigned long, std::function) (this=this@entry=0x7fdc1c21e800, kvID=, kvID@entry=37, seqNo=, seqNo@entry=4881, keyCallback=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:486 #7 0x00007fdc2383d420 in magma::Magma::Rollback(unsigned short, unsigned long, std::function) (this=this@entry=0x7fdc1c15e078, kvID=, seqNo=seqNo@entry=4881, keyCallback=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:516 #8 0x00007fdc2434846e in MagmaKVStore::rollback (this=this@entry=0x7fdc1c195e00, vbid=..., rollbackSeqno=rollbackSeqno@entry=4881, callback=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:2260 #9 0x00007fdc2420610c in EPBucket::doRollback (this=, vbid=..., rollbackSeqno=4881) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:1695 #10 0x00007fdc2429036c in KVBucket::rollback (this=0x7fdc1c1a0000, vbid=..., rollbackSeqno=4881) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2436 #11 0x00007fdc241b5fab in DcpConsumer::doRollback (this=0x7fdb905e9040, opaque=40, vbid=..., rollbackSeqno=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1126 #12 0x00007fdc241b6140 in RollbackTask::run (this=0x7fd981d66cd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:966 #13 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd981d66cd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #14 0x00007fdc24263782 in operator() (__closure=0x7fdb937e5040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #15 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #16 0x00007fdc243da470 in operator() (this=0x7fdb937e5040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #17 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc14061400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #18 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14061400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #19 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #20 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #21 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #22 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #23 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #24 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e69c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #25 operator() (__closure=0x7fdc1c3e69c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fdc24261c73 in operator() (this=0x7fdc1c40d2f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #28 operator() (__closure=0x7fdc1c40d2e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #29 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #30 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d31e0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #31 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #32 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fdb92ffd700 (LWP 46369)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fdc2429019c in __gthread_mutex_lock (__mutex=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 lock (this=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 #5 lock (this=0x7fdb92fe3ac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:267 #6 unique_lock (__m=..., this=0x7fdb92fe3ac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:197 #7 KVBucket::rollback (this=0x7fdc1c1a0000, vbid=..., rollbackSeqno=4876) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2414 #8 0x00007fdc241b5fab in DcpConsumer::doRollback (this=0x7fdb905e9040, opaque=59, vbid=..., rollbackSeqno=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1126 #9 0x00007fdc241b6140 in RollbackTask::run (this=0x7fd981d67b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:966 #10 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd981d67b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #11 0x00007fdc24263782 in operator() (__closure=0x7fdb92fe4040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #12 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #13 0x00007fdc243da470 in operator() (this=0x7fdb92fe4040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #14 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc14061400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #15 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14061400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #16 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #17 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #18 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #19 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #20 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #21 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6a00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fdc1c3e6a00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fdc24261c73 in operator() (this=0x7fdc1c40d340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #25 operator() (__closure=0x7fdc1c40d330) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3250) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #28 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fdb927fc700 (LWP 46370)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fdc2429019c in __gthread_mutex_lock (__mutex=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 lock (this=0x7fdc1c1a0130) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:103 #5 lock (this=0x7fdb927e2ac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:267 #6 unique_lock (__m=..., this=0x7fdb927e2ac0) at /usr/local/include/c++/7.3.0/bits/std_mutex.h:197 #7 KVBucket::rollback (this=0x7fdc1c1a0000, vbid=..., rollbackSeqno=4901) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2414 #8 0x00007fdc241b5fab in DcpConsumer::doRollback (this=0x7fdb905e9040, opaque=56, vbid=..., rollbackSeqno=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:1126 #9 0x00007fdc241b6140 in RollbackTask::run (this=0x7fd981d678d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/consumer.cc:966 #10 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd981d678d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #11 0x00007fdc24263782 in operator() (__closure=0x7fdb927e3040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #12 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #13 0x00007fdc243da470 in operator() (this=0x7fdb927e3040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #14 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc14061400, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #15 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc14061400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #16 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #17 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #18 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #19 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #20 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #21 0x00007fdc24261d98 in operator() (this=0x7fdc1c3e6980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #22 operator() (__closure=0x7fdc1c3e6980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #23 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #24 0x00007fdc24261c73 in operator() (this=0x7fdc1c40fc80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #25 operator() (__closure=0x7fdc1c40fc70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d32c0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #28 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fdb91ffb700 (LWP 46371)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc243c456a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fdbb27db798) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7fdbb27db798, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x00007fdc243c9e7d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7fdbb27db798, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x00007fdc243ca4e6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fdbc00039c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x00007fdc243cdc40 in try_wait_for > (timeout=, this=0x7fdbc00039c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fdbc0003980, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x00007fdc243c28d7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc1408e000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x00007fdc24261d98 in operator() (this=0x7fdc1401d080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #18 operator() (__closure=0x7fdc1401d080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fdc24261c73 in operator() (this=0x7fdc1c410db0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #21 operator() (__closure=0x7fdc1c410da0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #22 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #23 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3330) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #24 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fdb917fa700 (LWP 46372)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc24178c62 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=0x7fdb90471df8, state=@0x7fdb917e0b5c: 384, token=0x7fdb917e0b88, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/detail/Futex-inl.h:101 #4 0x00007fdc24178ca8 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=, token=, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:1298 #5 0x00007fdc242ca783 in ReadHolder (lock=..., this=0x7fdb917e0b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:419 #6 VBucket::processDurabilityTimeout (this=0x7fdb90471c00, asOf=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:677 #7 0x00007fdc24284a3b in VBCBAdaptor::run (this=0x7fdc1c26a610) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2313 #8 0x00007fdc24269343 in GlobalTask::execute (this=0x7fdc1c26a610) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #9 0x00007fdc24263782 in operator() (__closure=0x7fdb917e1040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #10 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #11 0x00007fdc243da470 in operator() (this=0x7fdb917e1040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #12 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc1408ec00, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #13 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc1408ec00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #14 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #15 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #16 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #17 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #18 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #19 0x00007fdc24261d98 in operator() (this=0x7fdc1401d180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #20 operator() (__closure=0x7fdc1401d180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #21 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #22 0x00007fdc24261c73 in operator() (this=0x7fdc1c410fe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #23 operator() (__closure=0x7fdc1c410fd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #24 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #25 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d33a0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #26 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #27 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fdb90ff9700 (LWP 46373)): #0 0x00007fdc1ffa8bf9 in syscall () from /lib64/libc.so.6 #1 0x00007fdc243bd35d in nativeFutexWaitImpl (waitMask=, absSteadyTime=, absSystemTime=, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=, absSteadyTime=, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x00007fdc24178c62 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=0x7fdb90471df8, state=@0x7fdb90fdfb5c: 385, token=0x7fdb90fdfb88, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/detail/Futex-inl.h:101 #4 0x00007fdc24178ca8 in folly::SharedMutexImpl::lockSharedImpl::WaitForever> (this=, token=, ctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:1298 #5 0x00007fdc242ca783 in ReadHolder (lock=..., this=0x7fdb90fdfb80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/SharedMutex.h:419 #6 VBucket::processDurabilityTimeout (this=0x7fdb90471c00, asOf=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:677 #7 0x00007fdc24284a3b in VBCBAdaptor::run (this=0x7fd8da5f5650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2313 #8 0x00007fdc24269343 in GlobalTask::execute (this=0x7fd8da5f5650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:74 #9 0x00007fdc24263782 in operator() (__closure=0x7fdb90fe0040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:195 #10 folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #11 0x00007fdc243da470 in operator() (this=0x7fdb90fe0040) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #12 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) ( this=this@entry=0x7fdc1408ec00, thread=..., task=task@entry=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #13 0x00007fdc243c294a in folly::CPUThreadPoolExecutor::threadRun (this=0x7fdc1408ec00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #14 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #15 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #16 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #17 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #18 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #19 0x00007fdc24261d98 in operator() (this=0x7fdc1401d1c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #20 operator() (__closure=0x7fdc1401d1c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:54 #21 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #22 0x00007fdc24261c73 in operator() (this=0x7fdc1c411030) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:416 #23 operator() (__closure=0x7fdc1c411020) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/executors/thread_factory/PriorityThreadFactory.h:54 #24 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #25 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3410) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #26 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #27 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fdb73fff700 (LWP 46374)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc238d4d89 in __gthread_cond_timedwait (__abs_timeout=0x7fdb73fe6150, __mutex=, __cond=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 magma::TaskQueue::dequeue (this=0x7fdc1c5d3480, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:97 #6 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #7 0x00007fdc22a4d727 in run (this=0x7fdc1c355290) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #8 platform_thread_wrap (arg=0x7fdc1c355290) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #9 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fdb737fe700 (LWP 46375)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc238d4d89 in __gthread_cond_timedwait (__abs_timeout=0x7fdb737e5150, __mutex=, __cond=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 magma::TaskQueue::dequeue (this=0x7fdc1c5d3480, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:97 #6 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #7 0x00007fdc22a4d727 in run (this=0x7fdc1c355350) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #8 platform_thread_wrap (arg=0x7fdc1c355350) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #9 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fdb72ffd700 (LWP 46376)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc238d4d89 in __gthread_cond_timedwait (__abs_timeout=0x7fdb72fe4150, __mutex=, __cond=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 magma::TaskQueue::dequeue (this=0x7fdc1c5d3480, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:97 #6 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #7 0x00007fdc22a4d727 in run (this=0x7fdc1c355380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #8 platform_thread_wrap (arg=0x7fdc1c355380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #9 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fdb727fc700 (LWP 46377)): #0 0x00007fdc20289de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc238d4d89 in __gthread_cond_timedwait (__abs_timeout=0x7fdb727e3150, __mutex=, __cond=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_for > (__rtime=..., __lock=..., this=0x7fdc1c5d3488) at /usr/local/include/c++/7.3.0/condition_variable:138 #5 magma::TaskQueue::dequeue (this=0x7fdc1c5d3480, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:97 #6 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d3c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #7 0x00007fdc22a4d727 in run (this=0x7fdc1c3553b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #8 platform_thread_wrap (arg=0x7fdc1c3553b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #9 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fdb71ffb700 (LWP 46378)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d400) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c3553e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c3553e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fdb717fa700 (LWP 46379)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355410) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355410) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fdb70ff9700 (LWP 46380)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d480) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fdb6bfff700 (LWP 46381)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d4c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355470) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355470) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fdb6b7fe700 (LWP 46382)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d580) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c3554a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c3554a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fdb6affd700 (LWP 46383)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d5c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c3554d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c3554d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fdb6a7fc700 (LWP 46384)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fdb69ffb700 (LWP 46385)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d640) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355530) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355530) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fdb697fa700 (LWP 46386)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355560) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355560) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fdb68ff9700 (LWP 46387)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d6c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355590) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355590) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fdb687f8700 (LWP 46388)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c3555c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c3555c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fdb67ff7700 (LWP 46389)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d740) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c3555f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c3555f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fdb677f6700 (LWP 46390)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d780) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355620) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355620) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fdb66ff5700 (LWP 46391)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d7c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fdb667f4700 (LWP 46392)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d800) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c355680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c355680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fdb65ff3700 (LWP 46393)): #0 0x00007fdc20289a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fdc20a671bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fdc1c5d34f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #2 std::condition_variable::wait (this=this@entry=0x7fdc1c5d34f8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00007fdc238d4cbc in magma::TaskQueue::dequeue (this=0x7fdc1c5d34f0, shutdown=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #4 0x00007fdc238d51c3 in magma::TaskWorker::loop (w=0x7fdc1401d840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:135 #5 0x00007fdc22a4d727 in run (this=0x7fdc1c3556b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7fdc1c3556b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fdb657f2700 (LWP 46394)): #0 0x00007fdc1ffaeeb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fdc21b203c5 in epoll_dispatch (base=0x7fdc14246000, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fdc21b17376 in event_base_loop (base=0x7fdc14246000, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x00007fdc243e27e7 in folly::EventBase::loopBody (this=this@entry=0x7fdc1419d780, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x00007fdc243e2cb6 in folly::EventBase::loop (this=this@entry=0x7fdc1419d780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x00007fdc243e4676 in folly::EventBase::loopForever (this=0x7fdc1419d780) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00007fdc243d1c99 in folly::IOThreadPoolExecutor::threadRun (this=0x7fdc1c5bb780, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x00007fdc243dd429 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007fdc20a6cdcf in std::execute_native_thread_routine (__p=0x7fdc1c5d3560) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #13 0x00007fdc20285ea5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fdc1ffae8dd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fdc2500a400 (LWP 45886)): #0 0x00007fdc2028c54d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fdc20287e9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fdc20287d68 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000420d4b in bucketsForEach(std::function, void*) (fn=..., arg=0x0) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748 #4 0x00000000004aa76e in mc_time_clock_tick () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mc_time.cc:259 #5 0x00000000004aa955 in mc_time_clock_event_handler (fd=, which=, arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mc_time.cc:198 #6 0x00007fdc21b16f58 in event_process_active_single_queue (base=base@entry=0x7fdc1f81f000, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1697 #7 0x00007fdc21b1754f in event_process_active (base=0x7fdc1f81f000) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #8 event_base_loop (base=0x7fdc1f81f000, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #9 0x0000000000427fca in memcached_main (argc=, argv=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1542 #10 0x00007fdc1fed2555 in __libc_start_main () from /lib64/libc.so.6 #11 0x00000000004203c5 in _start () Detaching from program: /opt/couchbase/bin/memcached, process 45886 [Inferior 1 (process 45886) detached]