Basic crash dump analysis of 3847df73.core. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxrwxrwx 1 couchbase couchbase 10540776 Sep 21 03:38 /opt/couchbase/bin/memcached 6f42e777233ace95f4bf331b17fda6d0 /opt/couchbase/bin/memcached -rw-r--r-- 1 root root 1581056 Sep 24 17:04 3847df73.core 3b2941d68c09a8bb812db6ebc488fd2b 3847df73.core -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...Reading symbols from /usr/lib/debug/opt/couchbase/bin/memcached.debug...done. done. Illegal process-id: 3847df73.core. [New LWP 27079] [New LWP 27057] [New LWP 27058] [New LWP 27059] [New LWP 27060] [New LWP 27061] [New LWP 27065] [New LWP 27066] [New LWP 27067] [New LWP 27068] [New LWP 27069] [New LWP 27070] [New LWP 27071] [New LWP 27072] [New LWP 27073] [New LWP 27074] [New LWP 27075] [New LWP 27076] [New LWP 27077] [New LWP 27078] [New LWP 27080] [New LWP 27081] [New LWP 27082] [New LWP 27083] [New LWP 27084] [New LWP 27085] [New LWP 27233] [New LWP 27234] [New LWP 27235] [New LWP 27236] [New LWP 27237] [New LWP 27238] [New LWP 27239] [New LWP 27240] [New LWP 27241] [New LWP 27242] [New LWP 27243] [New LWP 27244] [New LWP 27245] [New LWP 27246] [New LWP 27247] [New LWP 27248] [New LWP 27249] [New LWP 27250] [New LWP 27251] [New LWP 27252] [New LWP 27253] Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Failed to read a valid object file image from memory. Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fa7298775c9 in raise () from /lib64/libc.so.6 warning: File "/opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". Thread 47 (LWP 27253): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700780) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 46 (LWP 27252): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700750) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 45 (LWP 27251): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700720) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 44 (LWP 27250): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa7287006f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 43 (LWP 27249): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa7287006c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 42 (LWP 27248): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700690) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 41 (LWP 27247): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700660) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 40 (LWP 27246): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700630) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 39 (LWP 27245): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 38 (LWP 27244): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700570) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 37 (LWP 27243): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa7287005a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 36 (LWP 27242): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa7287005d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 35 (LWP 27241): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 34 (LWP 27240): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa7287004e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 33 (LWP 27239): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa728700510) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 32 (LWP 27238): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72a1e58bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009589f8 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:101 #4 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #5 0x0000000000b010e9 in run (this=0x7fa7287004b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 31 (LWP 27237): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7fa6e17e8960, __mutex=, __cond=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7fa728696ae0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 30 (LWP 27236): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7fa6e1fe9960, __mutex=, __cond=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7fa728696b70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 29 (LWP 27235): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7fa6e27ea960, __mutex=, __cond=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7fa728696b10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 28 (LWP 27234): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000958c8f in __gthread_cond_timedwait (__abs_timeout=0x7fa6e2feb960, __mutex=, __cond=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fa728736a58) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:99 #6 0x0000000000958de1 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #7 0x0000000000b010e9 in run (this=0x7fa728720090) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 27 (LWP 27233): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72b513e51 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fa6e37ed220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 26 (LWP 27085): #0 0x00007fa729938783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa72b2db3c5 in epoll_dispatch (base=0x7fa728593500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa72b2d2376 in event_base_loop (base=0x7fa728593500, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7fa72859bf00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7fa72859bf00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=0x7fa72859bf00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000b5b459 in folly::IOThreadPoolExecutor::threadRun (this=0x7fa72859b500, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 25 (LWP 27084): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=83785240, absSteadyTime=absSteadyTime@entry=0x7fa704fe75f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa704fe7618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d74b0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2591740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2591700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728677800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 24 (LWP 27083): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=92177944, absSteadyTime=absSteadyTime@entry=0x7fa7057e85f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa7057e8618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7420, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2591740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2591700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728677800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 23 (LWP 27082): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=100570648, absSteadyTime=absSteadyTime@entry=0x7fa705fe95f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa705fe9618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d70d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2591340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2591300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728676c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 22 (LWP 27081): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=108963352, absSteadyTime=absSteadyTime@entry=0x7fa7067ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa7067ea618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b53777 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7570, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2590ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2590a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728676000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 21 (LWP 27080): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=117356056, absSteadyTime=absSteadyTime@entry=0x7fa706feb5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa706feb618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7078, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2590ac0, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2590a80, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728676000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 20 (LWP 27078): #0 0x00007fa72c014f7d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fa72c010d41 in _L_lock_790 () from /lib64/libpthread.so.0 #2 0x00007fa72c010c47 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000aece4f in google_breakpad::ExceptionHandler::SignalHandler (sig=6, info=0x7fa707feb970, uc=0x7fa707feb840) at src/client/linux/handler/exception_handler.cc:342 #4 #5 0x00007fa7298775c9 in raise () from /lib64/libc.so.6 #6 0x00007fa729878cd8 in abort () from /lib64/libc.so.6 #7 0x00007fa72a1c37f2 in __gnu_cxx::__verbose_terminate_handler() () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:50 #8 0x0000000000aa9bcb in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 #9 0x00007fa72a1c18f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 #10 0x00007fa72a1c1961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 #11 0x00007fa72a1c1bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fa6f4000940, tinfo=0xfc9360 , dest=0x442e60 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 #12 0x00000000004fa3d3 in magma::BasicFile::Sync (this=0x7fa6ce67a240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:202 #13 0x00000000009a6af4 in magma::FileWithStats::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:63 #14 0x00000000009210bb in magma::WAL::sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:581 #15 0x0000000000922f6b in magma::WAL::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:539 #16 0x000000000090e05b in magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function >&)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:172 #17 0x00000000008fe535 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #18 0x00000000008fe6e2 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #19 0x000000000085d11f in MagmaMemoryTrackingProxy::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #20 0x00000000008475cc in MagmaKVStore::saveDocs(MagmaKVStoreTransactionContext&, VB::Commit&, kvstats_ctx&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #21 0x000000000084043f in MagmaKVStore::commit(std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:637 #22 0x00000000007ea105 in EPBucket::commit(KVStoreIface&, std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:929 #23 0x00000000007f1105 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:801 #24 0x00000000007f14ff in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:378 #25 0x00000000006cba20 in Flusher::flushVB (this=0x7fa71c0aec00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:285 #26 0x00000000006cc370 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:200 #27 0x0000000000a1c222 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 #28 0x0000000000a19345 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fa707fed840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 #29 0x0000000000b63c30 in operator() (this=0x7fa707fed840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #30 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa728676000, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #31 0x0000000000b4b9ea in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728676000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #32 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #33 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #34 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #35 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #36 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #37 0x0000000000a18fd4 in operator() (this=0x7fa7295ce280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #38 operator() (__closure=0x7fa7295ce280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #39 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #40 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #41 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #42 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 19 (LWP 27077): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=377398808, absSteadyTime=absSteadyTime@entry=0x7fa7167ea5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa7167ea618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7120, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2587740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2587700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa7295e0400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 18 (LWP 27076): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=385791512, absSteadyTime=absSteadyTime@entry=0x7fa716feb5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa716feb618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7018, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2587740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2587700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa7295e0400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 17 (LWP 27075): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=394184216, absSteadyTime=absSteadyTime@entry=0x7fa7177ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa7177ec618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7048, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2587740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2587700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa7295e0400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce0c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce0c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 16 (LWP 27074): #0 0x00007fa729932989 in syscall () from /lib64/libc.so.6 #1 0x0000000000b4370d in nativeFutexWaitImpl (waitMask=waitMask@entry=402576920, absSteadyTime=absSteadyTime@entry=0x7fa717fed5f8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa717fed618, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000b4d7da in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000b5363d in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa7147d7060, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000b53ca6 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=0x2587740, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000b57400 in folly::UnboundedBlockingQueue::try_take_for (this=0x2587700, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000b4b977 in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa7295e0400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000a18fd4 in operator() (this=0x7fa7295ce080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #15 operator() (__closure=0x7fa7295ce080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 15 (LWP 27073): #0 0x00007fa729938783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa72b2db3c5 in epoll_dispatch (base=0x7fa728593200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa72b2d2376 in event_base_loop (base=0x7fa728593200, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7fa7285f3380, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7fa7285f3380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7fa7285f3380) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7fa7285f3370) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7fa7295efe90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 14 (LWP 27072): #0 0x00007fa729938783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa72b2db3c5 in epoll_dispatch (base=0x7fa728592f00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa72b2d2376 in event_base_loop (base=0x7fa728592f00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7fa7285f27b0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7fa7285f27b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7fa7285f27b0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7fa7285f27a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7fa7295efe00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 13 (LWP 27071): #0 0x00007fa729938783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa72b2db3c5 in epoll_dispatch (base=0x7fa728592c00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa72b2d2376 in event_base_loop (base=0x7fa728592c00, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7fa7285f1be0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7fa7285f1be0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7fa7285f1be0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7fa7285f1bd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7fa7295efb90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 12 (LWP 27070): #0 0x00007fa729938783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa72b2db3c5 in epoll_dispatch (base=0x7fa728592900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa72b2d2376 in event_base_loop (base=0x7fa728592900, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7fa7285f1010, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7fa7285f1010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=this@entry=0x7fa7285f1010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005a2a39 in worker_libevent (arg=0x7fa7285f1000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:115 #7 0x0000000000b010e9 in run (this=0x7fa7295ef830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 11 (LWP 27069): #0 0x00007fa72992dbdd in poll () from /lib64/libc.so.6 #1 0x0000000000a40b48 in master_thread (thread_func_param=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 10 (LWP 27068): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a49bdb in worker_thread (thread_func_param=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 9 (LWP 27067): #0 0x00007fa72c012705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a49bdb in worker_thread (thread_func_param=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #3 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 8 (LWP 27066): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000000000066c517 in AuditImpl::consume_events() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000b010e9 in run (this=0x7fa7295efaa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #3 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #4 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 7 (LWP 27065): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005e40a6 in ExternalAuthManagerThread::run() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000b0e5f3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #3 0x0000000000b010e9 in run (this=0x7fa7295ef980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 6 (LWP 27061): #0 0x00007fa72992dbdd in poll () from /lib64/libc.so.6 #1 0x000000000059d6f1 in get_command (buffsize=80, buffer=0x7fa7271ea990 "") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000b010e9 in run (this=0x7fa7295ef5f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 5 (LWP 27060): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000acef7a in __gthread_cond_timedwait (__abs_timeout=0x7fa7279eb9b0, __mutex=, __cond=0x7fa7295f11f8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7fa7295f11f8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7fa7295f11f8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=0x7fa7295f11f8) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=0x7fa7295f11f8) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7fa7285ca048) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/periodic_worker-inl.h:25 #7 __invoke_impl&, std::chrono::seconds):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7fa7285ca048) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7fa7285ca048) at /usr/local/include/c++/7.3.0/thread:243 #11 std::thread::_State_impl&, std::chrono::seconds):: > > >::_M_run(void) (this=0x7fa7285ca040) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 4 (LWP 27059): #0 0x00007fa7298aeef3 in fwrite () from /lib64/libc.so.6 #1 0x0000000000acaa74 in spdlog::details::file_helper::write (this=0x7fa7295ef4d0, buf=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/file_helper-inl.h:91 #2 0x0000000000ab243c in custom_rotating_file_sink::sink_it_ (this=0x7fa72858b550, msg=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/logger/custom_rotating_file_sink.cc:81 #3 0x0000000000ab9bba in spdlog::sinks::base_sink::log (this=0x7fa72858b550, msg=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/sinks/base_sink-inl.h:29 #4 0x0000000000943eb3 in spdlog::sinks::dist_sink::sink_it_ (this=, msg=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/spdlog.exploded/include/spdlog/sinks/dist_sink.h:60 #5 0x0000000000ab9bba in spdlog::sinks::base_sink::log (this=0x7fa7295f1160, msg=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/sinks/base_sink-inl.h:29 #6 0x0000000000acc37c in spdlog::async_logger::backend_sink_it_ (this=0x7fa728581510, msg=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/async_logger-inl.h:63 #7 0x0000000000acf344 in spdlog::details::thread_pool::process_next_msg_ (this=this@entry=0x7fa7295fc110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:108 #8 0x0000000000acf508 in worker_loop_ (this=0x7fa7295fc110) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:90 #9 operator() (__closure=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:28 #10 __invoke_impl):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #11 __invoke):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #12 _M_invoke<0> (this=) at /usr/local/include/c++/7.3.0/thread:234 #13 operator() (this=) at /usr/local/include/c++/7.3.0/thread:243 #14 std::thread::_State_impl):: > > >::_M_run(void) (this=) at /usr/local/include/c++/7.3.0/thread:186 #15 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #16 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 3 (LWP 27058): #0 0x00007fa72c012ab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa72b5145a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7fa728fee220) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7fa728fee220) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa7299381ad in clone () from /lib64/libc.so.6 Thread 2 (LWP 27057): #0 0x00007fa729938783 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa72b2db3c5 in epoll_dispatch (base=0x7fa728592300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa72b2d2376 in event_base_loop (base=0x7fa728592300, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b6d317 in folly::EventBase::loopBody (this=this@entry=0x7fa72859b280, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b6d7e6 in folly::EventBase::loop (this=this@entry=0x7fa72859b280) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b6f1a6 in folly::EventBase::loopForever (this=0x7fa72859b280) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000559978 in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1019 #7 0x00007fa729863af5 in __libc_start_main () from /lib64/libc.so.6 #8 0x0000000000552141 in _start () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_construct.h:151 Thread 1 (LWP 27079): #0 0x00007fa7298775c9 in raise () from /lib64/libc.so.6 #1 0x00007fa729878cd8 in abort () from /lib64/libc.so.6 #2 0x00007fa72a1b663c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x0000000000aa9bcb in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 #4 0x00007fa72a1c18f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 #5 0x00007fa72a1c1961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 #6 0x00007fa72a1c1bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7fa6ec000940, tinfo=0xfc9360 , dest=0x442e60 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 #7 0x00000000004fa3d3 in magma::BasicFile::Sync (this=0x7fa6ceb30560) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:202 #8 0x00000000009a6af4 in magma::FileWithStats::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:63 #9 0x00000000009210bb in magma::WAL::sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:581 #10 0x0000000000922f6b in magma::WAL::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:539 #11 0x000000000090e05b in magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function >&)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:172 #12 0x00000000008fe535 in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #13 0x00000000008fe6e2 in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #14 0x000000000085d11f in MagmaMemoryTrackingProxy::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function >&)>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #15 0x00000000008475cc in MagmaKVStore::saveDocs(MagmaKVStoreTransactionContext&, VB::Commit&, kvstats_ctx&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #16 0x000000000084043f in MagmaKVStore::commit(std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:637 #17 0x00000000007ea105 in EPBucket::commit(KVStoreIface&, std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:929 #18 0x00000000007f1105 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:801 #19 0x00000000007f14ff in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:378 #20 0x00000000006cba20 in Flusher::flushVB (this=0x7fa71c0af300) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:285 #21 0x00000000006cc370 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:200 #22 0x0000000000a1c222 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:68 #23 0x0000000000a19345 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7fa7077ec840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:189 #24 0x0000000000b63c30 in operator() (this=0x7fa7077ec840) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #25 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa728676000, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #26 0x0000000000b4b9ea in folly::CPUThreadPoolExecutor::threadRun (this=0x7fa728676000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #27 0x0000000000b66be9 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #28 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #29 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #30 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #31 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #32 0x0000000000a18fd4 in operator() (this=0x7fa7295ce2c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #33 operator() (__closure=0x7fa7295ce2c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:47 #34 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #35 0x00007fa72a1ead40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #36 0x00007fa72c00edf3 in start_thread () from /lib64/libpthread.so.0 #37 0x00007fa7299381ad in clone () from /lib64/libc.so.6 -------------------------------------------------------------------------------- Module information: