2021-06-04 13:08:03,931 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:735] 172.23.106.217: 1319 core dump seen 2021-06-04 13:08:03,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.217: rm -rf /opt/couchbase/var/lib/couchbase/crash/bb344735-b68b-45ea-cba57bb8-fd6b2a29.core 2021-06-04 13:08:03,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.217: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/bb344735-b68b-45ea-cba57bb8-fd6b2a29.dmp > /opt/couchbase/var/lib/couchbase/crash/bb344735-b68b-45ea-cba57bb8-fd6b2a29.core 2021-06-04 13:08:03,979 | test | INFO | MainThread | [basetestcase:get_gdb:677] running gdb --batch /opt/couchbase/bin/ memcached -c /opt/couchbase/var/lib/couchbase/crash/bb344735-b68b-45ea-cba57bb8-fd6b2a29.core -ex "bt full" -ex quit 2021-06-04 13:08:03,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.217: gdb --batch /opt/couchbase/bin/ memcached -c /opt/couchbase/var/lib/couchbase/crash/bb344735-b68b-45ea-cba57bb8-fd6b2a29.core -ex "bt full" -ex quit 2021-06-04 13:08:04,114 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:736] 172.23.106.217: Stack Trace of first crash - bb344735-b68b-45ea-cba57bb8-fd6b2a29.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f100f86e337 in ?? () #0 0x00007f100f86e337 in ?? () No symbol table info available. #1 0x00007f100f86fa28 in ?? () No symbol table info available. #2 0x0000000000000020 in ?? () No symbol table info available. #3 0x0000000000000000 in ?? () No symbol table info available. 2021-06-04 13:08:04,114 | test | INFO | MainThread | [basetestcase:get_full_thread_dump:686] running gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2021-06-04 13:08:04,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.217: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 61 (Thread 0x7f81e31ff700 (LWP 22873)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e58415a7 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7f81e247e700 (LWP 22874)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a855a5 in spdlog::details::thread_pool::process_next_msg_() () #2 0x0000000000a85bc0 in spdlog::details::thread_pool::worker_loop_() () #3 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #4 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7f81e1c7d700 (LWP 22875)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a84d5b in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv () #2 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #3 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7f81e147c700 (LWP 22876)): #0 0x00007f81e3c5abed in poll () from /lib64/libc.so.6 #1 0x000000000058a841 in check_stdin_thread(void*) () #2 0x0000000000aab9e9 in platform_thread_wrap(void*) () #3 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7f81e0c7b700 (LWP 22877)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in worker_thread () #2 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7f81e047a700 (LWP 22878)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in worker_thread () #2 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7f81dfc79700 (LWP 22879)): #0 0x00007f81e3c5abed in poll () from /lib64/libc.so.6 #1 0x00000000008c0db8 in master_thread () #2 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7f81df478700 (LWP 22880)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005bb7d6 in ExternalAuthManagerThread::run() () #2 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #3 0x0000000000aab9e9 in platform_thread_wrap(void*) () #4 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7f81de8f7700 (LWP 22881)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000629ee7 in AuditImpl::consume_events() () #2 0x0000000000aab9e9 in platform_thread_wrap(void*) () #3 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7f81de0f6700 (LWP 22882)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000058adcb in worker_libevent(void*) () #4 0x0000000000aab9e9 in platform_thread_wrap(void*) () #5 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7f81dd8f5700 (LWP 22883)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000058adcb in worker_libevent(void*) () #4 0x0000000000aab9e9 in platform_thread_wrap(void*) () #5 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7f81dd0f4700 (LWP 22884)): #0 0x00007ffc263dc6c2 in clock_gettime () #1 0x00007f81e3c7b7ad in clock_gettime () from /lib64/libc.so.6 #2 0x00007f81e4514ca1 in std::chrono::_V2::steady_clock::now () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/chrono.cc:88 #3 0x00000000005c9583 in Cookie::initialize(cb::mcbp::Header const&, bool) () #4 0x00000000005af184 in Connection::executeCommandPipeline() () #5 0x00000000005b08e7 in Connection::executeCommandsCallback() () #6 0x00000000005b1501 in Connection::rw_callback(bufferevent*, void*) () #7 0x00007f81e55f5d8e in bufferevent_run_deferred_callbacks_unlocked () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #8 0x00007f81e55fed01 in event_process_active_single_queue.isra.31 () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #9 0x00007f81e55ff54f in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #10 0x000000000058adcb in worker_libevent(void*) () #11 0x0000000000aab9e9 in platform_thread_wrap(void*) () #12 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7f81dc8f3700 (LWP 22885)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000058adcb in worker_libevent(void*) () #4 0x0000000000aab9e9 in platform_thread_wrap(void*) () #5 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7f81d7fff700 (LWP 22886)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000058adcb in worker_libevent(void*) () #4 0x0000000000aab9e9 in platform_thread_wrap(void*) () #5 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7f81d77fe700 (LWP 22887)): #0 0x00007f81e3c5c340 in readv () from /lib64/libc.so.6 #1 0x00007f81e55f2bd6 in evbuffer_read () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55f9f44 in bufferevent_readcb () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x00007f81e55fefaa in event_process_active_single_queue.isra.31 () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #4 0x00007f81e55ff54f in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #5 0x000000000058adcb in worker_libevent(void*) () #6 0x0000000000aab9e9 in platform_thread_wrap(void*) () #7 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7f81d6ffd700 (LWP 22888)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000058adcb in worker_libevent(void*) () #4 0x0000000000aab9e9 in platform_thread_wrap(void*) () #5 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7f81d67fc700 (LWP 22889)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7f81d5ffb700 (LWP 22890)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f81d57fa700 (LWP 22891)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f81d4ff9700 (LWP 22892)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f81cffff700 (LWP 22893)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f81cf7fe700 (LWP 22894)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f81ceffd700 (LWP 22895)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f818ffff700 (LWP 22933)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9ef7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f818f7fe700 (LWP 22934)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f818effd700 (LWP 22935)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f818e7fc700 (LWP 22936)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e5840e51 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f818dffb700 (LWP 22937)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f818d7fa700 (LWP 22938)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f818cff9700 (LWP 22939)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f817a7da700 (LWP 22940)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f8179fd9700 (LWP 22941)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9dbd in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f81797d8700 (LWP 22942)): #0 0x00000000009b9912 in magma::Memtable::Add(magma::Operation, long, magma::Slice const&, magma::Slice const&) () #1 0x0000000000961a67 in magma::LSMTree::add(magma::Operation, magma::Slice const&, magma::Slice const&, bool) () #2 0x000000000096253e in magma::LSMTree::Write(magma::Operation, magma::Slice const&, magma::Slice const&) () #3 0x0000000000939fcd in magma::KVStore::writeDocument(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #4 0x000000000093a37c in magma::KVStore::Write(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #5 0x00000000009287e7 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #6 0x0000000000928c72 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #7 0x000000000084f1a7 in MagmaKVStore::saveDocs(VB::Commit&, kvstats_ctx&) () #8 0x0000000000854c30 in MagmaKVStore::commit(VB::Commit&) () #9 0x00000000007ee44e in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () #10 0x00000000007f379d in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () #11 0x00000000007f3b4f in EPBucket::flushVBucket(Vbid) () #12 0x000000000069d860 in Flusher::flushVB() () #13 0x000000000069e329 in Flusher::step(GlobalTask*) () #14 0x00000000006a0567 in GlobalTask::execute() () #15 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () #16 0x0000000000b0a3b0 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () #17 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #18 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #19 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #20 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f8178fd7700 (LWP 22943)): #0 0x00007f81e6107fe3 in pwrite64 () from /lib64/libpthread.so.0 #1 0x00000000009866f8 in magma::BasicFile::Write(long, char const*, unsigned long) () #2 0x00000000009a14aa in magma::WAL::flushBuffer() () #3 0x00000000009a2dce in magma::WAL::Flush() () #4 0x00000000009a1cc1 in magma::WAL::AddRecord(magma::WAL::walRecType, std::vector > const&, magma::WALOffset&) () #5 0x0000000000939d90 in magma::KVStore::writeDocument(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #6 0x000000000093a37c in magma::KVStore::Write(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #7 0x00000000009287e7 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #8 0x0000000000928c72 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #9 0x000000000084f1a7 in MagmaKVStore::saveDocs(VB::Commit&, kvstats_ctx&) () #10 0x0000000000854c30 in MagmaKVStore::commit(VB::Commit&) () #11 0x00000000007ee44e in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () #12 0x00000000007f379d in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () #13 0x00000000007f3b4f in EPBucket::flushVBucket(Vbid) () #14 0x000000000069d860 in Flusher::flushVB() () #15 0x000000000069e329 in Flusher::step(GlobalTask*) () #16 0x00000000006a0567 in GlobalTask::execute() () #17 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () #18 0x0000000000b0a3b0 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () #19 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #20 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #21 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #22 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #23 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f815ffff700 (LWP 22944)): #0 0x00007f81e6107fe3 in pwrite64 () from /lib64/libpthread.so.0 #1 0x00000000009866f8 in magma::BasicFile::Write(long, char const*, unsigned long) () #2 0x00000000009a14aa in magma::WAL::flushBuffer() () #3 0x00000000009a2dce in magma::WAL::Flush() () #4 0x00000000009a1cc1 in magma::WAL::AddRecord(magma::WAL::walRecType, std::vector > const&, magma::WALOffset&) () #5 0x0000000000939d90 in magma::KVStore::writeDocument(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #6 0x000000000093a37c in magma::KVStore::Write(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #7 0x00000000009287e7 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #8 0x0000000000928c72 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #9 0x000000000084f1a7 in MagmaKVStore::saveDocs(VB::Commit&, kvstats_ctx&) () #10 0x0000000000854c30 in MagmaKVStore::commit(VB::Commit&) () #11 0x00000000007ee44e in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () #12 0x00000000007f379d in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () #13 0x00000000007f3b4f in EPBucket::flushVBucket(Vbid) () #14 0x000000000069d860 in Flusher::flushVB() () #15 0x000000000069e329 in Flusher::step(GlobalTask*) () #16 0x00000000006a0567 in GlobalTask::execute() () #17 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () #18 0x0000000000b0a3b0 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () #19 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #20 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #21 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #22 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #23 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #24 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f815f7fe700 (LWP 22945)): #0 0x00000000009b9a35 in magma::Memtable::Add(magma::Operation, long, magma::Slice const&, magma::Slice const&) () #1 0x0000000000961a67 in magma::LSMTree::add(magma::Operation, magma::Slice const&, magma::Slice const&, bool) () #2 0x000000000096253e in magma::LSMTree::Write(magma::Operation, magma::Slice const&, magma::Slice const&) () #3 0x0000000000939fcd in magma::KVStore::writeDocument(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #4 0x000000000093a37c in magma::KVStore::Write(magma::Magma::WriteOperation const&, magma::LookupBuffer*, bool, std::function, magma::WAL*) () #5 0x00000000009287e7 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #6 0x0000000000928c72 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () #7 0x000000000084f1a7 in MagmaKVStore::saveDocs(VB::Commit&, kvstats_ctx&) () #8 0x0000000000854c30 in MagmaKVStore::commit(VB::Commit&) () #9 0x00000000007ee44e in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () #10 0x00000000007f379d in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () #11 0x00000000007f3b4f in EPBucket::flushVBucket(Vbid) () #12 0x000000000069d860 in Flusher::flushVB() () #13 0x000000000069e329 in Flusher::step(GlobalTask*) () #14 0x00000000006a0567 in GlobalTask::execute() () #15 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () #16 0x0000000000b0a3b0 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () #17 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #18 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #19 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #20 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f815effd700 (LWP 22946)): #0 0x000000000077e114 in Blob::New(char const*, unsigned long) () #1 0x00000000006aa6ef in Item::Item(DocKey const&, unsigned int, long, void const*, unsigned long, unsigned char, unsigned long, long, Vbid, unsigned long, unsigned char) () #2 0x0000000000842279 in MagmaKVStore::makeItem(Vbid, magma::Slice const&, magma::Slice const&, magma::Slice const&, ValueFilter) () #3 0x000000000084c210 in MagmaKVStore::scan(BySeqnoScanContext&) () #4 0x000000000088bb43 in DCPBackfillBySeqnoDisk::scan() () #5 0x000000000088e95a in DCPBackfillDisk::run() () #6 0x0000000000890ca1 in BackfillManager::backfill() () #7 0x0000000000890f14 in BackfillManagerTask::run() () #8 0x00000000006a0567 in GlobalTask::execute() () #9 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const () #10 0x0000000000b0a3b0 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () #11 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #12 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #13 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #14 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #15 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #16 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f815e7fc700 (LWP 22947)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9ef7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f815dffb700 (LWP 22948)): #0 0x00007f81e3c5fba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000af3f5a in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000af9ef7 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000afa426 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000699d4d in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f815d7fa700 (LWP 22949)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in magma::TaskQueue::dequeue(std::atomic&) () #2 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #3 0x0000000000aab9e9 in platform_thread_wrap(void*) () #4 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f815cff9700 (LWP 22950)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in magma::TaskQueue::dequeue(std::atomic&) () #2 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #3 0x0000000000aab9e9 in platform_thread_wrap(void*) () #4 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f8147fff700 (LWP 22951)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in magma::TaskQueue::dequeue(std::atomic&) () #2 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #3 0x0000000000aab9e9 in platform_thread_wrap(void*) () #4 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f81477fe700 (LWP 22952)): #0 0x00007f81e6104da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in magma::TaskQueue::dequeue(std::atomic&) () #2 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #3 0x0000000000aab9e9 in platform_thread_wrap(void*) () #4 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f8146ffd700 (LWP 22953)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f81467fc700 (LWP 22954)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f8145ffb700 (LWP 22955)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f81457fa700 (LWP 22956)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f813ffff700 (LWP 23022)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f813f7fe700 (LWP 23023)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f813effd700 (LWP 23024)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f813e7fc700 (LWP 23025)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f813afff700 (LWP 23026)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f813a7fe700 (LWP 23027)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f8139ffd700 (LWP 23028)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f81397fc700 (LWP 23029)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f8138ffb700 (LWP 23030)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f81387fa700 (LWP 23031)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f8137ff9700 (LWP 23032)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f81377f8700 (LWP 23033)): #0 0x00007f81e61049f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f81e45178bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () #5 0x0000000000aab9e9 in platform_thread_wrap(void*) () #6 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f8135bf7700 (LWP 23038)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b13a97 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b13f66 in folly::EventBase::loop() () #5 0x0000000000b15926 in folly::EventBase::loopForever() () #6 0x0000000000b01bd9 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b0d369 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x00007f81e451cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #9 0x00007f81e6100e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f81e3c6588d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f81e6facc40 (LWP 22872)): #0 0x00007f81e3c65e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f81e56083c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f81e55ff376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x000000000053bb26 in memcached_main(int, char**) () #4 0x00007f81e3b89505 in __libc_start_main () from /lib64/libc.so.6 #5 0x0000000000531d94 in _start () Detaching from program: /opt/couchbase/bin/memcached, process 22872