Basic crash dump analysis of 782420fc.core. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxrwxrwx 1 couchbase couchbase 10540776 Sep 21 03:38 /opt/couchbase/bin/memcached 6f42e777233ace95f4bf331b17fda6d0 /opt/couchbase/bin/memcached -rw-r--r-- 1 root root 1581056 Sep 24 16:59 782420fc.core 1912a6061dde5c486f2f09a635d2c817 782420fc.core -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...Reading symbols from /usr/lib/debug/opt/couchbase/bin/memcached.debug...done. done. Illegal process-id: 782420fc.core. [New LWP 26438] [New LWP 26413] [New LWP 26414] [New LWP 26415] [New LWP 26416] [New LWP 26417] [New LWP 26421] [New LWP 26422] [New LWP 26424] [New LWP 26425] [New LWP 26426] [New LWP 26427] [New LWP 26428] [New LWP 26429] [New LWP 26430] [New LWP 26431] [New LWP 26432] [New LWP 26433] [New LWP 26434] [New LWP 26435] [New LWP 26436] [New LWP 26437] [New LWP 26439] [New LWP 26440] [New LWP 26441] [New LWP 26442] [New LWP 26462] [New LWP 26463] [New LWP 26464] [New LWP 26469] [New LWP 26470] [New LWP 26475] [New LWP 26476] [New LWP 26477] [New LWP 26478] [New LWP 26487] [New LWP 26488] [New LWP 26489] [New LWP 26490] [New LWP 26491] [New LWP 26492] [New LWP 26493] [New LWP 26494] [New LWP 26495] [New LWP 26496] [New LWP 26497] [New LWP 26498] Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Failed to read a valid object file image from memory. Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f34428e25c9 in raise () from /lib64/libc.so.6 warning: File "/opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". Thread 47 (LWP 26498): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672960) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 46 (LWP 26497): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672930) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 45 (LWP 26496): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 44 (LWP 26495): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f34416728d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 43 (LWP 26494): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f34416728a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 42 (LWP 26493): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672870) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 41 (LWP 26492): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 40 (LWP 26491): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672810) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 39 (LWP 26490): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f34416727e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 38 (LWP 26489): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f34416727b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 37 (LWP 26488): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672780) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 36 (LWP 26487): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672750) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 35 (LWP 26478): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672720) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 34 (LWP 26477): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f34416726c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 33 (LWP 26476): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f3441672690) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 32 (LWP 26475): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f34432508bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7f34416726f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 31 (LWP 26470): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7f33f27ea960, __mutex=, __cond=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7f3441672630) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 30 (LWP 26469): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7f33f2feb960, __mutex=, __cond=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7f3441672600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 29 (LWP 26464): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7f33f37ec960, __mutex=, __cond=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7f34416725d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 28 (LWP 26463): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f344457ee51 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7f33f3fee220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 27 (LWP 26462): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7f3418bed960, __mutex=, __cond=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f344168ca58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7f34416a5bd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 26 (LWP 26442): #0 0x00007f34429a3783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f34443463c5 in epoll_dispatch (base=0x7f3441593500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f344433d376 in event_base_loop (base=0x7f3441593500, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7f344159bf00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7f344159bf00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=0x7f344159bf00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000b5b459 in folly::IOThreadPoolExecutor::threadRun (this=0x7f344159b500, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 25 (LWP 26441): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=436114968, absSteadyTime=absSteadyTime@entry=0x7f3419fe95f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f3419fe9618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9618, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e8e6c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e8e680, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441654800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 24 (LWP 26440): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=444507672, absSteadyTime=absSteadyTime@entry=0x7f341a7ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f341a7ea618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d95e8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e8e6c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e8e680, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441654800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 23 (LWP 26439): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=452900376, absSteadyTime=absSteadyTime@entry=0x7f341afeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f341afeb618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9120, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e8e2c0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e8e280, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441653c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 22 (LWP 26437): #0 0x00007f344507ff7d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f344507bd41 in _L_lock_790 () from /lib64/libpthread.so.0 #2 0x00007f344507bc47 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000aece4f in google_breakpad::ExceptionHandler::SignalHandler (sig=6, info=0x7f341bfeb530, uc=0x7f341bfeb400) at src/client/linux/handler/exception_handler.cc:342 #4 #5 0x00007f34428e25c9 in raise () from /lib64/libc.so.6 #6 0x00007f34428e3cd8 in abort () from /lib64/libc.so.6 #7 0x00007f344322e7f2 in __gnu_cxx::__verbose_terminate_handler() () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:50 #8 0x0000000000aa9bcb in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 #9 0x00007f344322c8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 #10 0x00007f344322c961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 #11 0x00007f344322cbf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f3410000940, tinfo=0xfc9360 , dest=0x442e60 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 #12 0x00000000004fa3d3 in magma::BasicFile::Sync (this=0x7f33e5ee8b50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:202 #13 0x00000000009a6af4 in magma::FileWithStats::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:63 #14 0x000000000091de96 in magma::WAL::setMaxSize(magma::WALOffset&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:1149 #15 0x0000000000922200 in magma::WAL::truncateSegment(magma::WALOffset, magma::WALOffset) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:1020 #16 0x000000000092268c in magma::WAL::fixFatalIOError() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:1183 #17 0x0000000000922a4d in magma::WAL::BeginTxn(magma::Slice&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:240 #18 0x000000000090d867 in magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function >&)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:124 #19 0x00000000008fe535 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #20 0x00000000008fe6e2 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #21 0x000000000085d11f in MagmaMemoryTrackingProxy::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #22 0x00000000008475cc in MagmaKVStore::saveDocs(MagmaKVStoreTransactionContext&, VB::Commit&, kvstats_ctx&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #23 0x000000000084043f in MagmaKVStore::commit(std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:637 #24 0x00000000007ea105 in EPBucket::commit(KVStoreIface&, std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:929 #25 0x00000000007f1105 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:801 #26 0x00000000007f14ff in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:378 #27 0x00000000006cba20 in Flusher::flushVB (this=0x7f342c286300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:285 #28 0x00000000006cc370 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:200 #29 0x0000000000a1c222 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 #30 0x0000000000a19345 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f341bfed840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 #31 0x0000000000b63c30 in operator() (this=0x7f341bfed840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #32 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7f3441653000, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #33 0x0000000000b4b9ea in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441653000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #34 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #35 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #36 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #37 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #38 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #39 0x0000000000a18fd4 in operator() (this=0x7f34425ce300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #40 operator() (__closure=0x7f34425ce300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #41 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #42 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #43 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #44 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 21 (LWP 26436): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=754734616, absSteadyTime=absSteadyTime@entry=0x7f342cfc55f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f342cfc5618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9528, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e8dac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e8da80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441653000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce2c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce2c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 20 (LWP 26435): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=763127320, absSteadyTime=absSteadyTime@entry=0x7f342d7c65f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f342d7c6618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9078, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e8dac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e8da80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441653000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 19 (LWP 26434): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=796837400, absSteadyTime=absSteadyTime@entry=0x7f342f7ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f342f7ec618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9048, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e84740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e84700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441626400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 18 (LWP 26433): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=805230104, absSteadyTime=absSteadyTime@entry=0x7f342ffed5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f342ffed618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9018, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e84740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e84700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441626400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 17 (LWP 26432): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=882820632, absSteadyTime=absSteadyTime@entry=0x7f34349ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f34349ec618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9060, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e84740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e84700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441626400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce0c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce0c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 16 (LWP 26431): #0 0x00007f344299d989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=891213336, absSteadyTime=absSteadyTime@entry=0x7f34351ed5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f34351ed618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7f342d7d9030, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2e84740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2e84700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441626400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7f34425ce080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7f34425ce080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 15 (LWP 26430): #0 0x00007f34429a3783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f34443463c5 in epoll_dispatch (base=0x7f3441593200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f344433d376 in event_base_loop (base=0x7f3441593200, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7f344162c380, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7f344162c380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7f344162c380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7f344162c370) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7f34425efe90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 14 (LWP 26429): #0 0x00007f34429a3783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f34443463c5 in epoll_dispatch (base=0x7f3441592f00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f344433d376 in event_base_loop (base=0x7f3441592f00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7f344162b7b0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7f344162b7b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7f344162b7b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7f344162b7a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7f34425efe00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 13 (LWP 26428): #0 0x00007f34429a3783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f34443463c5 in epoll_dispatch (base=0x7f3441592c00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f344433d376 in event_base_loop (base=0x7f3441592c00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7f344162abe0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7f344162abe0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7f344162abe0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7f344162abd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7f34425efb90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 12 (LWP 26427): #0 0x00007f34429a3783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f34443463c5 in epoll_dispatch (base=0x7f3441592900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f344433d376 in event_base_loop (base=0x7f3441592900, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7f344162a010, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7f344162a010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7f344162a010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7f344162a000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7f34425ef830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 11 (LWP 26426): #0 0x00007f3442998bdd in poll () from /lib64/libc.so.6 #1 0x0000000000a40b48 in master_thread (thread_func_param=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 10 (LWP 26425): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a49bdb in worker_thread (thread_func_param=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 9 (LWP 26424): #0 0x00007f344507d705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a49bdb in worker_thread (thread_func_param=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #3 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 8 (LWP 26422): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000066c517 in AuditImpl::consume_events() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000b010e9 in run (this=0x7f34425efaa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #3 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #4 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #5 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 7 (LWP 26421): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005e40a6 in ExternalAuthManagerThread::run() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000b0e5f3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #3 0x0000000000b010e9 in run (this=0x7f34425ef980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 6 (LWP 26417): #0 0x00007f3442998bdd in poll () from /lib64/libc.so.6 #1 0x000000000059d6f1 in get_command (buffsize=80, buffer=0x7f34401ea990 "") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000b010e9 in run (this=0x7f34425ef5f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 5 (LWP 26416): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000acef7a in __gthread_cond_timedwait (__abs_timeout=0x7f34409eb9b0, __mutex=, __cond=0x7f34425f11f8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f34425f11f8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7f34425f11f8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=0x7f34425f11f8) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=0x7f34425f11f8) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7f34415ca048) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/periodic_worker-inl.h:25 #7 __invoke_impl&, std::chrono::seconds):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7f34415ca048) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7f34415ca048) at /usr/local/include/c++/7.3.0/thread:243 #11 std::thread::_State_impl&, std::chrono::seconds):: > > >::_M_run(void) (this=0x7f34415ca040) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 4 (LWP 26415): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000acf1ba in __gthread_cond_timedwait (__abs_timeout=0x7f34411ec800, __mutex=, __cond=0x7f34425fc138) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f34425fc138) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7f34425fc138) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __atime=..., __lock=..., this=0x7f34425fc138) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7f34425fc138) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7f34425fc110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/mpmc_blocking_q.h:58 #7 spdlog::details::thread_pool::process_next_msg_ (this=this@entry=0x7f34425fc110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:99 #8 0x0000000000acf508 in worker_loop_ (this=0x7f34425fc110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:90 #9 operator() (__closure=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:28 #10 __invoke_impl):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #11 __invoke):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #12 _M_invoke<0> (this=) at /usr/local/include/c++/7.3.0/thread:234 #13 operator() (this=) at /usr/local/include/c++/7.3.0/thread:243 #14 std::thread::_State_impl):: > > >::_M_run(void) (this=) at /usr/local/include/c++/7.3.0/thread:186 #15 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #16 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 3 (LWP 26414): #0 0x00007f344507dab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f344457f5a7 in background_thread_sleep (tsdn=, interval=3800000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7f3441fee220) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7f3441fee220) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f34429a31ad in clone () from /lib64/libc.so.6 Thread 2 (LWP 26413): #0 0x00007f34429a3783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007f34443463c5 in epoll_dispatch (base=0x7f3441592300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f344433d376 in event_base_loop (base=0x7f3441592300, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7f344159b280, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7f344159b280) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=0x7f344159b280) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000559978 in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1019 #7 0x00007f34428ceaf5 in __libc_start_main () from /lib64/libc.so.6 #8 0x0000000000552141 in _start () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_construct.h:151 Thread 1 (LWP 26438): #0 0x00007f34428e25c9 in raise () from /lib64/libc.so.6 #1 0x00007f34428e3cd8 in abort () from /lib64/libc.so.6 #2 0x00007f344322163c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x0000000000aa9bcb in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 #4 0x00007f344322c8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 #5 0x00007f344322c961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 #6 0x00007f344322cbf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f3404000940, tinfo=0xfc9360 , dest=0x442e60 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 #7 0x00000000004fa3d3 in magma::BasicFile::Sync (this=0x7f341823ef60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:202 #8 0x00000000009a6af4 in magma::FileWithStats::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:63 #9 0x00000000009210bb in magma::WAL::sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:581 #10 0x0000000000922f6b in magma::WAL::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:539 #11 0x000000000090e05b in magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function >&)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:172 #12 0x00000000008fe535 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #13 0x00000000008fe6e2 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #14 0x000000000085d11f in MagmaMemoryTrackingProxy::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #15 0x00000000008475cc in MagmaKVStore::saveDocs(MagmaKVStoreTransactionContext&, VB::Commit&, kvstats_ctx&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #16 0x000000000084043f in MagmaKVStore::commit(std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:637 #17 0x00000000007ea105 in EPBucket::commit(KVStoreIface&, std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:929 #18 0x00000000007f1105 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:801 #19 0x00000000007f14ff in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:378 #20 0x00000000006cba20 in Flusher::flushVB (this=0x7f342c285c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:285 #21 0x00000000006cc370 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:200 #22 0x0000000000a1c222 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 #23 0x0000000000a19345 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f341b7ec840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 #24 0x0000000000b63c30 in operator() (this=0x7f341b7ec840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #25 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7f3441653000, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #26 0x0000000000b4b9ea in folly::CPUThreadPoolExecutor::threadRun (this=0x7f3441653000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #27 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #28 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #29 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #30 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #31 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #32 0x0000000000a18fd4 in operator() (this=0x7f34425ce380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #33 operator() (__closure=0x7f34425ce380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #34 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #35 0x00007f3443255d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #36 0x00007f3445079df3 in start_thread () from /lib64/libpthread.so.0 #37 0x00007f34429a31ad in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: