############################################################ # # Step 1: Create a 10 node cluster # ############################################################ 2021-09-06 09:44:02,759 | test | INFO | pool-3-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.120.170 | kv | 7.1.0-1231-enterprise | 1.34152457372 | Cluster node | | 172.23.121.115 | None | | | <--- IN --- | | 172.23.121.116 | None | | | <--- IN --- | | 172.23.121.123 | None | | | <--- IN --- | | 172.23.121.124 | None | | | <--- IN --- | | 172.23.121.126 | None | | | <--- IN --- | | 172.23.121.127 | None | | | <--- IN --- | | 172.23.121.128 | None | | | <--- IN --- | | 172.23.121.129 | None | | | <--- IN --- | | 172.23.121.130 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ ############################################################ # # Step 2: Create required buckets and collections. # ############################################################ 2021-09-06 09:45:05,426 | test | DEBUG | pool-3-thread-23 | [rest_client:rebalance:1434] Rebalance operation started 2021-09-06 09:45:05,428 | test | INFO | pool-3-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-1231-enterprise | 1.38295197385 | Cluster node | | 172.23.121.115 | kv | 7.1.0-1231-enterprise | 2.99559471366 | Cluster node | | 172.23.120.170 | kv | 7.1.0-1231-enterprise | 2.97866969582 | Cluster node | | 172.23.121.129 | kv | 7.1.0-1231-enterprise | 2.96594193792 | Cluster node | | 172.23.121.124 | kv | 7.1.0-1231-enterprise | 2.75922892781 | Cluster node | | 172.23.121.126 | kv | 7.1.0-1231-enterprise | 2.72040302267 | Cluster node | | 172.23.121.116 | kv | 7.1.0-1231-enterprise | 2.17911575765 | Cluster node | | 172.23.121.127 | kv | 7.1.0-1231-enterprise | 2.90492957746 | Cluster node | | 172.23.121.123 | kv | 7.1.0-1231-enterprise | 2.92007551919 | Cluster node | | 172.23.121.130 | kv | 7.1.0-1231-enterprise | 2.70951480781 | Cluster node | | 172.23.121.131 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.132 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.133 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.134 | ['index,n1ql'] | | | <--- IN --- | | 172.23.121.135 | ['index,n1ql'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ ############################################################ # # Step 1: Create 1000000 items sequentially # ############################################################ 2021-09-06 09:47:18,605 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.120.170 with username: root, password: couchbase 2021-09-06 09:47:18,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-09-06 09:47:18,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: ls /etc/ 2021-09-06 09:47:18,974 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3480] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-09-06 09:47:18,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: ls /cygdrive/c/Windows 2021-09-06 09:47:18,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: uname -m 2021-09-06 09:47:19,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: cat /proc/cpuinfo 2021-09-06 09:47:19,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: df -Thl 2021-09-06 09:47:19,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: cat /proc/meminfo 2021-09-06 09:47:19,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: hostname 2021-09-06 09:47:19,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: hostname -d 2021-09-06 09:47:19,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.120.170: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-09-06 09:47:19,160 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.120.170 ############################################################ # # Step 2: Create 1000000 items sequentially # ############################################################ 2021-09-06 10:57:21,069 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 19974777 | 80.58 GiB | 63.87 GiB | 70.37 GiB | 100 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+-----------+-----+ 2021-09-06 10:57:21,069 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 0 | 20000000 | 0-0 | 0-1000000 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ ############################################################ # # Step 2.1: Update 1000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # Step 3: Create 1000000 items sequentially # ############################################################ ############################################################ # # Step 3.1: Update 1000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # Step 4: Rebalance in with Loading of docs # ############################################################ 2021-09-06 15:42:57,765 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-1231-enterprise | 63.5186595583 | Cluster node | | 172.23.121.115 | kv | 7.1.0-1231-enterprise | 49.758822036 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-1231-enterprise | 8.37716484926 | Cluster node | | 172.23.121.135 | index, n1ql | 7.1.0-1231-enterprise | 18.0602006689 | Cluster node | | 172.23.120.170 | kv | 7.1.0-1231-enterprise | 46.6913643332 | Cluster node | | 172.23.121.129 | kv | 7.1.0-1231-enterprise | 48.0020361415 | Cluster node | | 172.23.121.124 | kv | 7.1.0-1231-enterprise | 46.6870930678 | Cluster node | | 172.23.121.126 | kv | 7.1.0-1231-enterprise | 48.1358951521 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-1231-enterprise | 10.8801642289 | Cluster node | | 172.23.121.116 | kv | 7.1.0-1231-enterprise | 42.0438334846 | Cluster node | | 172.23.121.127 | kv | 7.1.0-1231-enterprise | 48.1698017285 | Cluster node | | 172.23.121.123 | kv | 7.1.0-1231-enterprise | 47.6220752798 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-1231-enterprise | 9.54961930572 | Cluster node | | 172.23.121.130 | kv | 7.1.0-1231-enterprise | 48.3390607102 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-1231-enterprise | 10.770814567 | Cluster node | | 172.23.121.140 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ ############################################################ # # Step 5: Rebalance Out with Loading of docs # ############################################################ 2021-09-06 16:47:59,220 | test | INFO | pool-3-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.121.128 | kv | 7.1.0-1231-enterprise | 21.1008008135 | Cluster node | | 172.23.121.140 | kv | 7.1.0-1231-enterprise | 20.5397109605 | Cluster node | | 172.23.121.115 | kv | 7.1.0-1231-enterprise | 20.5871139916 | Cluster node | | 172.23.121.132 | index, n1ql | 7.1.0-1231-enterprise | 5.6824012314 | Cluster node | | 172.23.121.135 | index, n1ql | 7.1.0-1231-enterprise | 7.5910543131 | Cluster node | | 172.23.120.170 | kv | 7.1.0-1231-enterprise | 20.2135774218 | Cluster node | | 172.23.121.129 | kv | 7.1.0-1231-enterprise | 20.1958539997 | Cluster node | | 172.23.121.124 | kv | 7.1.0-1231-enterprise | 21.2248535778 | Cluster node | | 172.23.121.126 | kv | 7.1.0-1231-enterprise | 21.0325047801 | Cluster node | | 172.23.121.134 | index, n1ql | 7.1.0-1231-enterprise | 5.9064702114 | Cluster node | | 172.23.121.116 | kv | 7.1.0-1231-enterprise | 20.7108156482 | Cluster node | | 172.23.121.127 | kv | 7.1.0-1231-enterprise | 20.898561792 | --- OUT ---> | | 172.23.121.123 | kv | 7.1.0-1231-enterprise | 21.389489757 | Cluster node | | 172.23.121.131 | index, n1ql | 7.1.0-1231-enterprise | 6.30515411178 | Cluster node | | 172.23.121.130 | kv | 7.1.0-1231-enterprise | 20.8820538892 | Cluster node | | 172.23.121.133 | index, n1ql | 7.1.0-1231-enterprise | 6.09538383066 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-09-06 17:15:31,716 | test | INFO | MainThread | [rest_client:rebalance_reached:117] Rebalance reached >100% in 15.0820000172 seconds 2021-09-06 17:15:33,266 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 40005508 | 80.58 GiB | 63.73 GiB | 155.60 GiB | 59.2859088497 | +-----------------+-----------+-----------------+----------+------------+-----+----------+-----------+-----------+------------+---------------+ 2021-09-06 17:15:33,266 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+------------------+------------------+-----------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+------------------+------------------+-----------------+---------------+ | 40000000 | 380000000 | 2000000-20000000 | 2000000-20000000 | 1000000-2000000 | 0-0 | +---------------+---------------+------------------+------------------+-----------------+---------------+ ############################################################ # # Step 6: Run Crash Recovery 20 times # ############################################################ 11th Iteration: ############################## 172.23.121.140: 1 core dump seen 2021-09-06 17:56:06,585 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: rm -rf /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.core running: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.dmp > /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.core 2021-09-06 17:56:06,612 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.dmp > /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.core running: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.core -ex "bt full" -ex quit 2021-09-06 17:56:06,737 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.core -ex "bt full" -ex quit 172.23.121.140: Stack Trace of first crash - 957f4b96-e0b0-4f33-cff63ba5-6b98720a.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fb55d5fd387 in raise () from /lib64/libc.so.6 #0 0x00007fb55d5fd387 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fb55d5fea78 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007fb55df4863c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 terminating = false t = #3 0x0000000000aaeeeb in backtrace_terminate_handler() () No symbol table info available. #4 0x00007fb55df538f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 No locals. #5 0x00007fb55df53961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 No locals. #6 0x00007fb55df53bf4 in __cxxabiv1::__cxa_throw (obj=, tinfo=0xfcd360 , dest=0x443090 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 globals = header = 0x7fb3180008c0 #7 0x00000000004d5305 in MagmaKVStore::makeCompactionContext(Vbid) [clone .cold] () No symbol table info available. #8 0x000000000085e934 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) () No symbol table info available. #9 0x000000000091528d in magma::DocSeqGCContext::Transform(magma::Slice const&, magma::Slice const&, std::__cxx11::basic_string, std::allocator >&, magma::Slice*) () No symbol table info available. #10 0x00000000009a1075 in magma::ItemGCFilter::Process(std::vector, std::allocator > >&, std::vector, std::allocator > >*) () No symbol table info available. #11 0x000000000093dc32 in magma::LSMTree::mergeSSTables(std::vector, std::allocator > >&, magma::ItemGCFilter&, unsigned long, double, std::vector, std::allocator > >*) () No symbol table info available. #12 0x0000000000999ee3 in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock&, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>) () No symbol table info available. #13 0x000000000099c93a in magma::LSMTree::compactLevel(int, std::unique_lock&, bool, std::function > (bool)>) () No symbol table info available. #14 0x000000000099cbcc in magma::LSMTree::compact(std::unique_lock&) () No symbol table info available. #15 0x0000000000930a98 in std::_Function_handler::_M_invoke(std::_Any_data const&) () No symbol table info available. #16 0x000000000096dcfd in magma::TimedTask::Complete(bool) () No symbol table info available. #17 0x000000000096f4f1 in magma::TaskWorker::loop(void*) () No symbol table info available. #18 0x0000000000b0c529 in platform_thread_wrap(void*) () No symbol table info available. #19 0x00007fb55fd9dea5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #20 0x00007fb55d6c58dd in clone () from /lib64/libc.so.6 No symbol table info available. ############################## running: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2021-09-06 17:56:09,700 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/liblz4.so.1...Reading symbols from /opt/couchbase/bin/../lib/liblz4.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/liblz4.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 54 (Thread 0x7f66479ff700 (LWP 78403)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6649efa5a7 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7f66468fe700 (LWP 78404)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000ad947a in spdlog::details::thread_pool::process_next_msg_() () #2 0x0000000000ad97c8 in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details11thread_poolC4EmmSt8functionIFvvEEEUlvE_EEEEE6_M_runEv () #3 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #4 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7f66460fd700 (LWP 78405)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000ad9248 in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv () #2 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #3 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7f66458fc700 (LWP 78406)): #0 0x00007f664830ec3d in poll () from /lib64/libc.so.6 #1 0x00000000005a1281 in check_stdin_thread(void*) () #2 0x0000000000b0c529 in platform_thread_wrap(void*) () #3 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7f663f7fe700 (LWP 78410)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005e4126 in ExternalAuthManagerThread::run() () #2 0x0000000000b1add3 in Couchbase::Thread::thread_entry() () #3 0x0000000000b0c529 in platform_thread_wrap(void*) () #4 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7f663effd700 (LWP 78411)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000066e5e7 in AuditImpl::consume_events() () #2 0x0000000000b0c529 in platform_thread_wrap(void*) () #3 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7f663ffff700 (LWP 78412)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a4523b in worker_thread () #2 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7f66448fa700 (LWP 78413)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a4523b in worker_thread () #2 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7f66450fb700 (LWP 78414)): #0 0x00007f664830ec3d in poll () from /lib64/libc.so.6 #1 0x0000000000a3c1a8 in master_thread () #2 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7f663e7fc700 (LWP 78415)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7f663dffb700 (LWP 78416)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7f663d7fa700 (LWP 78417)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7f663cff9700 (LWP 78418)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7f6637fff700 (LWP 78419)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7f66377fe700 (LWP 78420)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7f6636ffd700 (LWP 78421)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x00000000005a5d39 in worker_libevent(void*) () #7 0x0000000000b0c529 in platform_thread_wrap(void*) () #8 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #9 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7f6634bff700 (LWP 78422)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7f66343fe700 (LWP 78423)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7f6633bfd700 (LWP 78424)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7f6631bd7700 (LWP 78425)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7f66313d6700 (LWP 78426)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7f6630bd5700 (LWP 78427)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7f662bfff700 (LWP 78428)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7f662b7fe700 (LWP 78429)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7f662affd700 (LWP 78430)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7f662a7fc700 (LWP 78431)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7f6629ffb700 (LWP 78432)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7f66297fa700 (LWP 78433)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5fe1d in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7f6628ff9700 (LWP 78434)): #0 0x00007f6649eedcd5 in je_sdallocx_default () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #1 0x00000000005b6356 in cb_sized_free () #2 0x000000000084b6a0 in MagmaKVStore::scan(BySeqnoScanContext&) const () #3 0x000000000088937a in DCPBackfillBySeqnoDisk::scan() () #4 0x000000000088b9ea in DCPBackfillDisk::run() () #5 0x000000000088d521 in BackfillManager::backfill() () #6 0x000000000088db04 in BackfillManagerTask::run() () #7 0x0000000000a15ae2 in GlobalTask::execute() () #8 0x0000000000a12c05 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const () #9 0x0000000000b70410 in folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) () #10 0x0000000000b581ca in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #11 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #12 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #13 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #14 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #15 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7f65fbfff700 (LWP 78435)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5ff57 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7f65fb7fe700 (LWP 78436)): #0 0x00007f6648313bf9 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4feed in folly::detail::futexWaitImpl(std::atomic const*, unsigned int, std::chrono::time_point > > const*, std::chrono::time_point > > const*, unsigned int) () #2 0x0000000000b59fba in folly::detail::FutexResult folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > >(std::atomic const*, unsigned int, std::chrono::time_point > > const&, unsigned int) () #3 0x0000000000b5ff57 in bool folly::SaturatingSemaphore::tryWaitSlow > >(std::chrono::time_point > > const&, folly::WaitOptions const&) () #4 0x0000000000b60486 in bool folly::detail::LifoSemBase, std::atomic>::try_wait_until > >(std::chrono::time_point > > const&) () #5 0x0000000000b63be0 in folly::UnboundedBlockingQueue::try_take_for(std::chrono::duration >) () #6 0x0000000000b58157 in folly::CPUThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x0000000000a12784 in void folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) () #9 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7f65faffd700 (LWP 78437)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x0000000000b67c39 in folly::IOThreadPoolExecutor::threadRun(std::shared_ptr) () #7 0x0000000000b733c9 in void folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) () #8 0x00007f6648bd0d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #9 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7f65fa7fc700 (LWP 78438)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6649ef9e51 in background_thread_entry () from /opt/couchbase/bin/../lib/libjemalloc.so.2 #2 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7f65f9ffb700 (LWP 78439)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000096f38f in magma::TaskQueue::dequeue(std::atomic&) () #2 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #3 0x0000000000b0c529 in platform_thread_wrap(void*) () #4 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7f65f97fa700 (LWP 78440)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000096f38f in magma::TaskQueue::dequeue(std::atomic&) () #2 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #3 0x0000000000b0c529 in platform_thread_wrap(void*) () #4 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7f65f8ff9700 (LWP 78441)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000096f38f in magma::TaskQueue::dequeue(std::atomic&) () #2 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #3 0x0000000000b0c529 in platform_thread_wrap(void*) () #4 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7f65dffff700 (LWP 78442)): #0 0x00007f664a9f5de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000096f38f in magma::TaskQueue::dequeue(std::atomic&) () #2 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #3 0x0000000000b0c529 in platform_thread_wrap(void*) () #4 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7f65df7fe700 (LWP 78443)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f65deffd700 (LWP 78444)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f65de7fc700 (LWP 78445)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f65ddffb700 (LWP 78446)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f65dd7fa700 (LWP 78448)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f65d79ff700 (LWP 78449)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f65d71fe700 (LWP 78450)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f65d69fd700 (LWP 78451)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f65d61fc700 (LWP 78452)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f65d59fb700 (LWP 78453)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f65d51fa700 (LWP 78454)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f65d49f9700 (LWP 78455)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f65c3fff700 (LWP 78456)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f65c37fe700 (LWP 78457)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f65c2ffd700 (LWP 78458)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f65c27fc700 (LWP 78459)): #0 0x00007f664a9f5a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f6648bcb8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000096f0f8 in magma::TaskQueue::dequeue(std::atomic&) () #4 0x000000000096f4e1 in magma::TaskWorker::loop(void*) () #5 0x0000000000b0c529 in platform_thread_wrap(void*) () #6 0x00007f664a9f1ea5 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f66483198dd in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f664b868c40 (LWP 78402)): #0 0x00007f6648319eb3 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f6649cc13c5 in epoll_dispatch () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #2 0x00007f6649cb8376 in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 #3 0x0000000000b79af7 in folly::EventBase::loopBody(int, bool) () #4 0x0000000000b79fc6 in folly::EventBase::loop() () #5 0x0000000000b7b986 in folly::EventBase::loopForever() () #6 0x000000000055a8b8 in memcached_main(int, char**) () #7 0x00007f664823d555 in __libc_start_main () from /lib64/libc.so.6 #8 0x00000000005531d1 in _start () Detaching from program: /opt/couchbase/bin/memcached, process 78402 [Inferior 1 (process 78402) detached] 2021-09-06 17:56:10,302 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-09-06 17:56:10,328 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-09-06 17:56:10,512 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-09-06 17:56:10,549 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-09-06 17:56:10,572 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-09-06 17:56:10,592 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-09-06 17:56:10,608 | test | DEBUG | Thread-13725 | [basetestcase:check_coredump_exist:854] 172.23.121.140: No 'sanitizers.log.*' files found 2021-09-06 17:56:10,608 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-09-06 17:56:10,625 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-09-06 17:56:10,762 | test | INFO | Thread-13725 | [basetestcase:check_logs:794] Comparing timestamps: Log's latest timestamp: 2021-09-06 17:52:57, Test's start timestamp is 2021-09-06 09:37:57.426000 2021-09-06 17:56:10,765 | test | INFO | Thread-13725 | [basetestcase:check_logs:794] Comparing timestamps: Log's latest timestamp: 2021-09-06 17:52:57, Test's start timestamp is 2021-09-06 09:37:57.426000 2021-09-06 17:56:10,765 | test | INFO | Thread-13725 | [basetestcase:check_coredump_exist:903] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-09-06 17:56:10,766 | test | CRITICAL | Thread-13725 | [basetestcase:check_coredump_exist:905] 172.23.121.140: Found ' CRITICAL ' logs - ['[ns_server:info,2021-09-06T17:52:56.117-07:00,babysitter_of_ns_1@cb.local:<0.19463.3>:ns_port_server:log:221]memcached<0.19463.3>: 2021-09-06T17:52:55.916371-07:00 CRITICAL *** Fatal error encountered during exception handling ***\n', 'memcached<0.19463.3>: 2021-09-06T17:52:55.916501-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): MagmaKVStore::makeCompactionContext vb:57 failed to read vbstate from disk. Status:NotFound: MagmaKVStore::readLocalDoc vb:57 key:_vbstate not found.\n', '[ns_server:info,2021-09-06T17:52:56.930-07:00,babysitter_of_ns_1@cb.local:<0.19463.3>:ns_port_server:log:221]memcached<0.19463.3>: CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1231). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.dmp before terminating.\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.560924-07:00 CRITICAL Detected previous crash\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.560986-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-1231). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/957f4b96-e0b0-4f33-cff63ba5-6b98720a.dmp before terminating.\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.560997-07:00 CRITICAL Stack backtrace of crashed thread:\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.560998-07:00 CRITICAL #0 /opt/couchbase/bin/memcached() [0x400000+0x69e8c8]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561000-07:00 CRITICAL #1 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x6f6e1a]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561002-07:00 CRITICAL #2 /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x6f7158]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561003-07:00 CRITICAL #3 /lib64/libpthread.so.0() [0x7fb55fd96000+0xf630]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561005-07:00 CRITICAL #4 /lib64/libc.so.6(gsignal+0x37) [0x7fb55d5c7000+0x36387]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561006-07:00 CRITICAL #5 /lib64/libc.so.6(abort+0x148) [0x7fb55d5c7000+0x37a78]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561008-07:00 CRITICAL #6 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb55deaf000+0x9963c]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561009-07:00 CRITICAL #7 /opt/couchbase/bin/memcached() [0x400000+0x6aeeeb]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561011-07:00 CRITICAL #8 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb55deaf000+0xa48f6]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561011-07:00 CRITICAL #9 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb55deaf000+0xa4961]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561013-07:00 CRITICAL #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fb55deaf000+0xa4bf4]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561041-07:00 CRITICAL #11 /opt/couchbase/bin/memcached() [0x400000+0xd5305]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561043-07:00 CRITICAL #12 /opt/couchbase/bin/memcached() [0x400000+0x45e934]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561044-07:00 CRITICAL #13 /opt/couchbase/bin/memcached() [0x400000+0x51528d]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561045-07:00 CRITICAL #14 /opt/couchbase/bin/memcached() [0x400000+0x5a1075]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561047-07:00 CRITICAL #15 /opt/couchbase/bin/memcached() [0x400000+0x53dc32]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561048-07:00 CRITICAL #16 /opt/couchbase/bin/memcached() [0x400000+0x599ee3]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561049-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x59c93a]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561050-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x59cbcc]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561051-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x530a98]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561052-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x56dcfd]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561082-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x56f4f1]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561084-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x70c529]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561110-07:00 CRITICAL #23 /lib64/libpthread.so.0() [0x7fb55fd96000+0x7ea5]\n', 'memcached<0.24404.3>: 2021-09-06T17:52:57.561112-07:00 CRITICAL #24 /lib64/libc.so.6(clone+0x6d) [0x7fb55d5c7000+0xfe8dd]\n', ' {<<"2021-09-06T17:52:57.561112-07:00 CRITICAL #24 /lib64/libc.so.6(clone+0x6d) [0x7fb55d5c7000+0xfe8dd]">>,\n', ' {<<"2021-09-06T17:52:57.561110-07:00 CRITICAL #23 /lib64/libpthread.so.0() [0x7fb55fd96000+0x7ea5]">>,\n', ' {<<"2021-09-06T17:52:57.561084-07:00 CRITICAL #22 /opt/couchbase/bin/memcached() [0x400000+0x70c529]">>,\n', ' {<<"2021-09-06T17:52:57.561082-07:00 CRITICAL #21 /opt/couchbase/bin/memcached() [0x400000+0x56f4f1]">>,\n', ' {<<"2021-09-06T17:52:57.561052-07:00 CRITICAL #20 /opt/couchbase/bin/memcached() [0x400000+0x56dcfd]">>,\n', ' {<<"2021-09-06T17:52:57.561051-07:00 CRITICAL #19 /opt/couchbase/bin/memcached() [0x400000+0x530a98]">>,\n', ' [{<<"2021-09-06T17:52:57.561049-07:00 CRITICAL #17 /opt/couchbase/bin/memcached() [0x400000+0x59c93a]">>,\n', ' {<<"2021-09-06T17:52:57.561050-07:00 CRITICAL #18 /opt/couchbase/bin/memcached() [0x400000+0x59cbcc]">>,\n'] 2021-09-06 17:56:10,766 | infra | DEBUG | Thread-13725 | [remote_util:execute_command_raw_jsch:3298] Running command on 172.23.121.140: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-09-06 17:56:10,783 | test | DEBUG | Thread-13725 | [basetestcase:check_coredump_exist:854] 172.23.121.140: No 'indexer.log*' files found 2021-09-06 17:56:10,783 | infra | DEBUG | Thread-13725 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.121.140