GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...Reading symbols from /usr/lib/debug/opt/couchbase/bin/memcached-7.2.1-5919.x86_64.debug...done. done. [New LWP 44026] [New LWP 43964] [New LWP 43970] [New LWP 43975] [New LWP 43976] [New LWP 43977] [New LWP 43981] [New LWP 43982] [New LWP 43985] [New LWP 43986] [New LWP 43987] [New LWP 43988] [New LWP 43989] [New LWP 43990] [New LWP 43991] [New LWP 43992] [New LWP 43993] [New LWP 43994] [New LWP 43995] [New LWP 43996] [New LWP 43997] [New LWP 43998] [New LWP 43999] [New LWP 44000] [New LWP 44001] [New LWP 44002] [New LWP 44003] [New LWP 44004] [New LWP 44005] [New LWP 44006] [New LWP 44007] [New LWP 44008] [New LWP 44009] [New LWP 44010] [New LWP 44011] [New LWP 44012] [New LWP 44013] [New LWP 44014] [New LWP 44015] [New LWP 44016] [New LWP 44017] [New LWP 44018] [New LWP 44019] [New LWP 44020] [New LWP 44021] [New LWP 44022] [New LWP 44023] [New LWP 44024] [New LWP 44025] [New LWP 44027] [New LWP 45990] [New LWP 45991] [New LWP 45992] [New LWP 45993] [New LWP 45994] [New LWP 45995] [New LWP 45996] [New LWP 45997] [New LWP 45998] [New LWP 45999] [New LWP 46000] [New LWP 46001] [New LWP 46002] [New LWP 46003] [New LWP 46004] [New LWP 46005] [New LWP 46006] [New LWP 46007] [New LWP 46008] [New LWP 46009] [New LWP 46010] [New LWP 46058] [New LWP 46779] To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f56b22ed387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Thread 73 (LWP 46779): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f56b3f81e11 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7f55ccdee220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007f56b4a6dea5 in start_thread (arg=0x7f55ccdff700) at pthread_create.c:307 #6 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 72 (LWP 46058): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f56b3f81e11 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7f55d1dee220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d1dff700) at pthread_create.c:307 #6 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 71 (LWP 46010): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d88400) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d3ff7700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 70 (LWP 46009): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d883a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d47f8700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 69 (LWP 46008): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d88220) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d4ff9700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 68 (LWP 46007): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d881f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d57fa700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 67 (LWP 46006): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d880a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d5ffb700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 66 (LWP 46005): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d88040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d67fc700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 65 (LWP 46004): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87f50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d6ffd700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 64 (LWP 46003): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87f20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d77fe700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 63 (LWP 46002): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87ef0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55d7fff700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 62 (LWP 46001): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87e90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f4ff9700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 61 (LWP 46000): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87da0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f57fa700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 60 (LWP 45999): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f5ffb700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 59 (LWP 45998): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f67fc700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 58 (LWP 45997): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87d10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f6ffd700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 57 (LWP 45996): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87cb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f77fe700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 56 (LWP 45995): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009ed518 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:105 #4 0x00000000009ed901 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:179 #5 0x0000000000baf779 in run (this=0x7f56b0d87c20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007f56b4a6dea5 in start_thread (arg=0x7f55f7fff700) at pthread_create.c:307 #8 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 55 (LWP 45994): #0 0x00000000009e8c4c in set_next (next=, n=) at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:100 #1 init (this_node=0x7f55bbecc1e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/boost.exploded/include/boost/intrusive/circular_list_algorithms.hpp:73 #2 generic_hook (this=0x7f55bbecc1e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/boost.exploded/include/boost/intrusive/detail/generic_hook.hpp:176 #3 list_base_hook (this=0x7f55bbecc1e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/boost.exploded/include/boost/intrusive/list_hook.hpp:84 #4 ObjectHandleImpl (object=..., id=13970343, this=0x7f55bbecc1d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/cache.h:101 #5 make_unique::ObjectHandleImpl, unsigned long&, std::shared_ptr&> () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/unique_ptr.h:962 #6 doCache (itr=..., itr=..., object=, id=13970343, this=0x7f5670de1f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/cache.h:415 #7 magma::Cache::AcquireObject (this=0x7f5670de1f40, ctor=..., doCaching=doCaching@entry=true) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/cache.h:322 #8 0x00000000009e1de0 in AcquireObject (doCaching=true, ctor=..., this=0x113ce40 ) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:275 #9 AcquireFile (file=0x7f55bf293580, this=0x113ce40 ) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:207 #10 magma::ManagedFile::Open(int) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:48 #11 0x0000000000a4d744 in magma::SSTable::Open(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:139 #12 0x0000000000a2ab1f in magma::LSMTree::writeSSTable(magma::SSTableWriter&, magma::Iterator*, long, std::function) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:766 #13 0x0000000000a2affc in magma::LSMTree::writeSSTable(bool, magma::Iterator*, unsigned long, long, bool, std::function) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:711 #14 0x0000000000a2b67e in magma::LSMTree::writeMemtable(magma::Memtable*, bool)::{lambda(magma::Iterator*)#4}::operator()(magma::Iterator*) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/memtable/memtable.h:69 #15 0x0000000000a2bec3 in magma::LSMTree::writeMemtable(magma::Memtable*, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:504 #16 0x0000000000a30de3 in magma::LSMTree::doMemtableFlushWork() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:545 #17 0x0000000000a317cd in operator() (__closure=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:929 #18 __invoke_impl, magma::LSMTree::newFlush()::&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #19 __invoke_r, magma::LSMTree::newFlush()::&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:115 #20 std::_Function_handler(), magma::LSMTree::newFlush():: >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:292 #21 0x0000000000a152bf in operator() (this=0x7f55bf4ee230) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #22 magma::FlushWork::Execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:61 #23 0x00000000009bd87f in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode)::{lambda()#2}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:515 #24 0x00000000009bf2d1 in __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #25 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #26 std::_Function_handler >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #27 0x00000000009ebf98 in operator() (this=0x7f55be62b8a0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #28 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158 #29 0x00000000009ed911 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:180 #30 0x0000000000baf779 in run (this=0x7f56b0d87aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #31 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5610ff9700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 54 (LWP 45993): #0 0x00000000009dd323 in FileWithStats (samplingInterval=, statsList=, makeFile=..., path=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #1 make_unique, std::allocator >&, const std::function >(const std::__cxx11::basic_string, std::allocator >&)>&, std::vector >, std::allocator > > >* const&, long unsigned int const&> () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/unique_ptr.h:962 #2 operator() (path=..., __closure=0x7f55becfc280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_system.cc:51 #3 __invoke_impl >, magma::FileSystemWithHistogram::FileSystemWithHistogram(magma::FileSystem, std::vector >*, uint64_t)::&, const std::__cxx11::basic_string, std::allocator >&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #4 __invoke_r, magma::FileSystemWithHistogram::FileSystemWithHistogram(magma::FileSystem, std::vector >*, uint64_t)::&, const std::__cxx11::basic_string, std::allocator >&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:113 #5 _ZNSt17_Function_handlerIFSt10unique_ptrIN5magma4FileESt14default_deleteIS2_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEZNS1_23FileSystemWithHistogramC4ENS1_10FileSystemEPSt6vectorIS0_INS1_14MagmaFileStatsES3_ISI_EESaISK_EEmEUlSD_E_E9_M_invokeERKSt9_Any_dataSD_ () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:292 #6 0x00000000009e0f5f in operator() (__args#0=..., this=0x7f55b206f988) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #7 magma::Cache::makeObject(magma::ManagedFile&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:248 #8 0x00000000009e89f9 in magma::Cache::AcquireObject (this=0x7f5670de2080, ctor=..., doCaching=doCaching@entry=true) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779 #9 0x00000000009e1de0 in AcquireObject (doCaching=true, ctor=..., this=0x113ce40 ) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:275 #10 AcquireFile (file=0x7f55b206f980, this=0x113ce40 ) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:207 #11 magma::ManagedFile::Open(int) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:48 #12 0x0000000000a4d744 in magma::SSTable::Open(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:139 #13 0x0000000000a2ab1f in magma::LSMTree::writeSSTable(magma::SSTableWriter&, magma::Iterator*, long, std::function) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:766 #14 0x0000000000a2ae94 in magma::LSMTree::writeSSTable(bool, magma::Iterator*, unsigned long, long, bool, std::function) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #15 0x0000000000a2b67e in magma::LSMTree::writeMemtable(magma::Memtable*, bool)::{lambda(magma::Iterator*)#4}::operator()(magma::Iterator*) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/memtable/memtable.h:69 #16 0x0000000000a2be84 in magma::LSMTree::writeMemtable(magma::Memtable*, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:494 #17 0x0000000000a30de3 in magma::LSMTree::doMemtableFlushWork() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:545 #18 0x0000000000a317cd in operator() (__closure=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:929 #19 __invoke_impl, magma::LSMTree::newFlush()::&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #20 __invoke_r, magma::LSMTree::newFlush()::&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:115 #21 std::_Function_handler(), magma::LSMTree::newFlush():: >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:292 #22 0x0000000000a152bf in operator() (this=0x7f55bf4eda10) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #23 magma::FlushWork::Execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:61 #24 0x00000000009bd928 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode)::{lambda()#2}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:522 #25 0x00000000009bf2d1 in __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #26 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #27 std::_Function_handler >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #28 0x00000000009ebf98 in operator() (this=0x7f55bf38dd60) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #29 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158 #30 0x00000000009ed911 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:180 #31 0x0000000000baf779 in run (this=0x7f56b0d87b90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #32 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #33 0x00007f56b4a6dea5 in start_thread (arg=0x7f56117fa700) at pthread_create.c:307 #34 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 53 (LWP 45992): #0 0x000000000057c81d in _Alloc_hider (__a=, __dat=, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #1 basic_string (__str=..., this=0x7f55b244b010) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:450 #2 pair (this=0x7f55b244b010) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_pair.h:314 #3 construct, nlohmann::basic_json<> >, const std::pair, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > > >&> (__p=0x7f55b244b010, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/ext/new_allocator.h:150 #4 construct, nlohmann::basic_json<> >, const std::pair, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > > >&> (__p=0x7f55b244b010, __a=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/alloc_traits.h:512 #5 _M_construct_node, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > > const&> (this=, __node=0x7f55b244aff0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:618 #6 _M_create_node, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > > const&> (this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:635 #7 operator(), nlohmann::basic_json<> >&> (__arg=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:552 #8 _M_clone_node, std::pair, nlohmann::basic_json<> >, std::_Select1st, nlohmann::basic_json<> > >, std::less, std::allocator, nlohmann::basic_json<> > > >::_Alloc_node> (this=, __node_gen=..., __x=0x7f55a748d430) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:662 #9 std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > >, std::_Select1st, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > > >, std::less, std::allocator, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > > > >::_M_copy, std::allocator >, std::pair, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > >, std::_Select1st, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > > >, std::less, std::allocator, std::allocator > const, nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > > > > >::_Alloc_node> (__x=0x7f55a748d430, __p=0x7f55be699de0, __p@entry=0x7f55bf02ed18, __node_gen=..., this=0x7f55bf02ed10) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:1893 #10 0x000000000057c3e7 in _M_copy, std::pair, nlohmann::basic_json<> >, std::_Select1st, nlohmann::basic_json<> > >, std::less, std::allocator, nlohmann::basic_json<> > > >::_Alloc_node> (__gen=, __x=..., this=0x7f55bf02ed10) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:211 #11 _M_copy (__x=..., this=0x7f55bf02ed10) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:908 #12 _Rb_tree (__x=..., this=0x7f55bf02ed10) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_tree.h:946 #13 map (this=0x7f55bf02ed10) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_map.h:207 #14 construct, nlohmann::basic_json<>, std::less, std::allocator, nlohmann::basic_json<> > > >, const std::map, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > >, std::less, std::allocator, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > > > > >&> (__p=0x7f55bf02ed10, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/ext/new_allocator.h:150 #15 construct, nlohmann::basic_json<>, std::less, std::allocator, nlohmann::basic_json<> > > >, const std::map, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > >, std::less, std::allocator, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > > > > >&> (__p=0x7f55bf02ed10, __a=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/alloc_traits.h:512 #16 create, nlohmann::basic_json<>, std::less, std::allocator, nlohmann::basic_json<> > > >, const std::map, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > >, std::less, std::allocator, std::allocator >, nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > > > > >&> () at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/json.hpp:928 #17 json_value (value=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/json.hpp:1077 #18 nlohmann::basic_json, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > >::basic_json (this=0x7f55bf0d2cf0, other=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/json.hpp:2058 #19 0x0000000000a59492 in construct, const nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > >&> (__p=, this=0x7f55bf045200) at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #20 construct, const nlohmann::basic_json, std::allocator >, bool, long int, long unsigned int, double, std::allocator, nlohmann::adl_serializer, std::vector > >&> (__p=, __a=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/alloc_traits.h:512 #21 push_back (__x=..., this=0x7f55bf045200) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_vector.h:1192 #22 push_back (val=..., this=0x7f5611fe90e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/json.hpp:5277 #23 magma::TreeSnapshot::encode[abi:cxx11]() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/encoding.cc:72 #24 0x0000000000a5ad36 in magma::TreeSnapshot::Write() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/encoding.cc:31 #25 0x0000000000a3e47f in magma::CheckpointManager::prepareSnapshot(magma::TreeSnapshot&, unsigned long, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:296 #26 0x0000000000a3ecc0 in magma::CheckpointTransaction::CreateSnapshot(unsigned long, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:529 #27 0x0000000000a153d1 in magma::FlushWork::CreateCheckpoint(unsigned long, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:76 #28 0x00000000009bdd87 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode)::{lambda()#2}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:549 #29 0x00000000009bf2d1 in __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #30 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #31 std::_Function_handler >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #32 0x00000000009ebf98 in operator() (this=0x7f55bf8204d0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #33 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158 #34 0x00000000009ed911 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:180 #35 0x0000000000baf779 in run (this=0x7f56b0d87a70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #36 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #37 0x00007f56b4a6dea5 in start_thread (arg=0x7f5611ffb700) at pthread_create.c:307 #38 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 52 (LWP 45991): #0 __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:1854 #1 0x0000000000a6ad0a in std::__cxx11::basic_string, std::allocator >::_M_append(char const*, unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:237 #2 0x0000000000a47b2c in append (__s=0xcc30ba ".data", this=0x7f56127e9cc0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:357 #3 operator+, std::allocator > (__rhs=0xcc30ba ".data", __lhs=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:6135 #4 magma::sstableFilePath[abi:cxx11](magma::SSTableConfig const&, unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:41 #5 0x0000000000a47d5b in getFilePath (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:685 #6 operator() (__closure=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:129 #7 __invoke_impl, magma::SSTable::Open(magma::ReaderStats&)::&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #8 __invoke_r, magma::SSTable::Open(magma::ReaderStats&)::&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:115 #9 std::_Function_handler, std::allocator >(), magma::SSTable::Open(magma::ReaderStats&):: >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:292 #10 0x00000000009e0f45 in operator() (this=0x7f55b56d8748) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #11 magma::Cache::makeObject(magma::ManagedFile&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:248 #12 0x00000000009e89f9 in magma::Cache::AcquireObject (this=0x7f5670de21c0, ctor=..., doCaching=doCaching@entry=true) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779 #13 0x00000000009e1de0 in AcquireObject (doCaching=true, ctor=..., this=0x113ce40 ) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:275 #14 AcquireFile (file=0x7f55b56d8720, this=0x113ce40 ) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:207 #15 magma::ManagedFile::Open(int) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:48 #16 0x0000000000a4d744 in magma::SSTable::Open(magma::ReaderStats&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:139 #17 0x0000000000a2ab1f in magma::LSMTree::writeSSTable(magma::SSTableWriter&, magma::Iterator*, long, std::function) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:766 #18 0x0000000000a2ae94 in magma::LSMTree::writeSSTable(bool, magma::Iterator*, unsigned long, long, bool, std::function) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #19 0x0000000000a2b67e in magma::LSMTree::writeMemtable(magma::Memtable*, bool)::{lambda(magma::Iterator*)#4}::operator()(magma::Iterator*) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/memtable/memtable.h:69 #20 0x0000000000a2be84 in magma::LSMTree::writeMemtable(magma::Memtable*, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:494 #21 0x0000000000a30de3 in magma::LSMTree::doMemtableFlushWork() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:545 #22 0x0000000000a317cd in operator() (__closure=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:929 #23 __invoke_impl, magma::LSMTree::newFlush()::&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #24 __invoke_r, magma::LSMTree::newFlush()::&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:115 #25 std::_Function_handler(), magma::LSMTree::newFlush():: >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:292 #26 0x0000000000a152bf in operator() (this=0x7f55a2202150) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #27 magma::FlushWork::Execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:61 #28 0x00000000009bd928 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode)::{lambda()#2}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:522 #29 0x00000000009bf2d1 in __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #30 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #31 std::_Function_handler >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #32 0x00000000009ebf98 in operator() (this=0x7f55b2c9e6f0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #33 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:158 #34 0x00000000009ed911 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:180 #35 0x0000000000baf779 in run (this=0x7f56b0d87260) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #36 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #37 0x00007f56b4a6dea5 in start_thread (arg=0x7f56127fc700) at pthread_create.c:307 #38 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 51 (LWP 45990): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f56b3f81e11 in background_thread_sleep (tsdn=, interval=7525000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7f5612fec220) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007f56b4a6dea5 in start_thread (arg=0x7f5612ffd700) at pthread_create.c:307 #6 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 50 (LWP 44027): #0 0x00007f56b23b60e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f56b3d493b5 in epoll_dispatch (base=0x7f56b0c8ce00, tv=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f56b3d40366 in event_base_loop (base=0x7f56b0c8ce00, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b11e6f00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b11e6f00) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c2cb26 in folly::EventBase::loopForever (this=0x7f56b11e6f00) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000c18dd9 in folly::IOThreadPoolExecutor::threadRun (this=0x7f56b11e6780, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007f56b4a6dea5 in start_thread (arg=0x7f56137fe700) at pthread_create.c:307 #14 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 49 (LWP 44025): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=821982744, absSteadyTime=absSteadyTime@entry=0x7f5630fe75f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdce30) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdce30, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f5630fe7618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdce30) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdce30, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdce30, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf5f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf5f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf5f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46d00, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b11f21c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b11f21c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f5630ff9700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 48 (LWP 44024): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=830375448, absSteadyTime=absSteadyTime@entry=0x7f56317e85f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbda8c8) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbda8c8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f56317e8618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbda8c8) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbda8c8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbda8c8, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf5f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf5f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf5f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46d00, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b11f2180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b11f2180) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f56317fa700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 47 (LWP 44023): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=838768152, absSteadyTime=absSteadyTime@entry=0x7f5631fe95f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdb4c8) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdb4c8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f5631fe9618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdb4c8) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdb4c8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdb4c8, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf5f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf5f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf5f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46d00, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b11f2140) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b11f2140) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f5631ffb700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 46 (LWP 44022): #0 0x00007f56b23acd7d in fdatasync () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000009dcbbf in magma::BasicFile::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:192 #2 0x0000000000a46750 in magma::FileWithStats::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:60 #3 0x00000000009de32b in magma::WriteFile(magma::FileSystem, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:127 #4 0x0000000000a5affd in magma::TreeSnapshot::Write() () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #5 0x0000000000a3e47f in magma::CheckpointManager::prepareSnapshot(magma::TreeSnapshot&, unsigned long, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:296 #6 0x0000000000a3ecc0 in magma::CheckpointTransaction::CreateSnapshot(unsigned long, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:529 #7 0x0000000000a153d1 in magma::FlushWork::CreateCheckpoint(unsigned long, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:76 #8 0x00000000009bdd87 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode)::{lambda()#2}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:549 #9 0x00000000009beb4b in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:593 #10 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #11 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #12 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #13 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #14 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #15 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #16 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #17 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #18 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #19 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #20 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #21 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #22 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56327ea650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #23 0x0000000000ac344e in operator() (this=0x7f56327ea650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #24 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #25 0x0000000000c215b0 in operator() (this=0x7f56327ea840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #26 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #27 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #28 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #29 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #30 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #31 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #32 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #33 0x0000000000abb474 in operator() (this=0x7f56b11f2000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #34 operator() (__closure=0x7f56b11f2000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #35 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #36 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #37 0x00007f56b4a6dea5 in start_thread (arg=0x7f56327fc700) at pthread_create.c:307 #38 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 45 (LWP 44021): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55d00e2b40) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55d00e2b40, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5632feb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5632feb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5632feb840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11e2fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11e2fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5632ffd700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 44 (LWP 44020): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c6663b40) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c6663b40, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56337ec650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56337ec650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56337ec840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11e2f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11e2f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56337fe700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 43 (LWP 44019): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c8750340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c8750340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5633fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5633fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5633fed840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11e2f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11e2f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5633fff700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 42 (LWP 44018): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c8a94340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c8a94340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5654fe7650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5654fe7650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5654fe7840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11e2f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11e2f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5654ff9700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 41 (LWP 44017): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c8750340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c8750340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56557e8650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56557e8650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56557e8840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11e2ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11e2ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56557fa700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 40 (LWP 44016): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c8750340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c8750340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5655fe9650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5655fe9650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5655fe9840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11e2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11e2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5655ffb700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 39 (LWP 44015): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c6663b40) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c6663b40, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56567ea650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56567ea650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56567ea840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6e40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6e40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56567fc700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 38 (LWP 44014): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c3a40340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c3a40340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5656feb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5656feb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5656feb840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5656ffd700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 37 (LWP 44013): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55d00e2b40) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55d00e2b40, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56577ec650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56577ec650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56577ec840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56577fe700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 36 (LWP 44012): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c8750340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c8750340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5657fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5657fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5657fed840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5657fff700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 35 (LWP 44011): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c8750340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c8750340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56717e8650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56717e8650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56717e8840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6d80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6d80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56717fa700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 34 (LWP 44010): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55d289fb40) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55d289fb40, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5671fe9650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5671fe9650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5671fe9840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5671ffb700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 33 (LWP 44009): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55d289fb40) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55d289fb40, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56727ea650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56727ea650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56727ea840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56727fc700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 32 (LWP 44008): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c641c340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c641c340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5672feb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f5672feb650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f5672feb840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f5672ffd700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 31 (LWP 44007): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f56b2c678bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f55c152d340) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7f55c152d340, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000009b860f in magma::KVStore::acquireFlushBarrier() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:1759 #4 0x00000000009be036 in magma::KVStore::flushMemTables(magma::WAL*, magma::WALOffset, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:410 #5 0x00000000009becd8 in magma::KVStore::FlushMemTables (this=, wal=, flushMode=, blockMode=blockMode@entry=magma::Blocking) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:394 #6 0x000000000095c835 in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1464 #7 0x000000000095c9fa in SyncKVStore (kvID=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:347 #8 magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:351 #9 0x00000000008b902d in MagmaMemoryTrackingProxy::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:489 #10 0x000000000089ad58 in MagmaKVStore::makeFileHandle(Vbid) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x000000000089f796 in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource, std::unique_ptr >) const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.cc:1688 #12 0x00000000008f2ae0 in DCPBackfillBySeqnoDisk::create() () at /opt/gcc-10.2.0/include/c++/10.2.0/ext/atomicity.h:70 #13 0x00000000008f5f32 in DCPBackfillDisk::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_disk.cc:151 #14 0x00000000008f7f13 in BackfillManager::backfill() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:345 #15 0x00000000008f81a4 in BackfillManagerTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:78 #16 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #17 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56737ec650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #18 0x0000000000ac344e in operator() (this=0x7f56737ec650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 0x0000000000c215b0 in operator() (this=0x7f56737ec840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #21 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46100, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #22 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46100, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #23 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #24 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #25 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #26 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #27 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #28 0x0000000000abb474 in operator() (this=0x7f56b11d6c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 operator() (__closure=0x7f56b11d6c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #31 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56737fe700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 30 (LWP 44006): #0 fetch_add (__m=std::memory_order_relaxed, __i=1, this=0x7f55a006c590) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:1090 #1 fetch_add (arg=1, this=0x7f55a006c590) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/relaxed_atomic.h:50 #2 operator++ (this=0x7f55a006c590) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/relaxed_atomic.h:87 #3 magma::LSMTree::Get(magma::Slice const&, std::shared_ptr&, magma::LookupBuffer&, magma::Slice&, bool&, bool, magma::ReaderStats*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:1090 #4 0x00000000009b8cc4 in magma::KVStore::KeyExists(magma::Slice const&, magma::LookupBuffer&, magma::ReaderStats*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:650 #5 0x00000000009c8778 in operator() (buffer=..., index=0, __closure=0x7f5673fec410) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:87 #6 magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function > const*> ()>, magma::Magma::HistoryMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:130 #7 0x00000000009866ac in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function > const*> ()>, magma::Magma::HistoryMode) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #8 0x00000000009868ff in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function > const*> ()>, magma::Magma::HistoryMode) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #9 0x00000000008b91c3 in MagmaMemoryTrackingProxy::WriteDocs(unsigned short, std::vector > const&, unsigned int, magma::Magma::HistoryMode, std::function, std::function > const*> ()>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #10 0x00000000008abf70 in MagmaKVStore::saveDocs(MagmaKVStoreTransactionContext&, VB::Commit&, kvstats_ctx&, magma::Magma::HistoryMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #11 0x00000000008947c5 in MagmaKVStore::commit(std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.h:808 #12 0x000000000083ab65 in EPBucket::commit(KVStoreIface&, std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:964 #13 0x00000000008427f9 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:836 #14 0x00000000008432ef in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:381 #15 0x00000000006ee2c6 in Flusher::flushVB (this=0x7f55cf5ee700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:293 #16 0x00000000006eec80 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:203 #17 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #18 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5673fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #19 0x0000000000ac344e in operator() (this=0x7f5673fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #21 0x0000000000c215b0 in operator() (this=0x7f5673fed840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #22 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b11df500, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #23 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11df500, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #24 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #25 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #26 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #27 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #28 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #29 0x0000000000abb474 in operator() (this=0x7f56b11d6b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 operator() (__closure=0x7f56b11d6b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #31 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #32 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #33 0x00007f56b4a6dea5 in start_thread (arg=0x7f5673fff700) at pthread_create.c:307 #34 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 29 (LWP 44005): #0 0x00007f56b4a75023 in pwrite64 () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000009d9fa6 in magma::BasicFile::Write(long, char const*, unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:175 #2 0x0000000000a465bc in magma::FileWithStats::Write(long, char const*, unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_stats.cc:52 #3 0x00000000009abb0d in magma::WAL::flushBuffer() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:748 #4 0x00000000009ac0f6 in magma::WAL::flush() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:464 #5 0x00000000009ac66a in magma::WAL::sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:495 #6 0x00000000009ae2c2 in magma::WAL::endTxn(magma::WAL::walRecType, magma::Slice&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:267 #7 0x00000000009ae3c5 in magma::WAL::EndTxn(magma::Slice&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/wal/wal.cc:277 #8 0x00000000009c8d6e in magma::KVStore::WriteDocs(magma::WAL*, std::vector > const&, std::function, std::function > const*> ()>, magma::Magma::HistoryMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/write.cc:206 #9 0x00000000009866ac in magma::Magma::Impl::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function > const*> ()>, magma::Magma::HistoryMode) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #10 0x00000000009868ff in magma::Magma::WriteDocs(unsigned short, std::vector > const&, unsigned int, std::function, std::function > const*> ()>, magma::Magma::HistoryMode) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #11 0x00000000008b91c3 in MagmaMemoryTrackingProxy::WriteDocs(unsigned short, std::vector > const&, unsigned int, magma::Magma::HistoryMode, std::function, std::function > const*> ()>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #12 0x00000000008abf70 in MagmaKVStore::saveDocs(MagmaKVStoreTransactionContext&, VB::Commit&, kvstats_ctx&, magma::Magma::HistoryMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #13 0x00000000008947c5 in MagmaKVStore::commit(std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-kvstore.h:808 #14 0x000000000083ab65 in EPBucket::commit(KVStoreIface&, std::unique_ptr >, VB::Commit&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:964 #15 0x00000000008427f9 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:836 #16 0x00000000008432ef in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:381 #17 0x00000000006ee2c6 in Flusher::flushVB (this=0x7f55cf5efc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:293 #18 0x00000000006eec80 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:203 #19 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #20 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5698bbf650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #21 0x0000000000ac344e in operator() (this=0x7f5698bbf650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #22 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #23 0x0000000000c215b0 in operator() (this=0x7f5698bbf840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #24 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b11df500, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #25 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11df500, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #26 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #27 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #28 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #29 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #30 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #31 0x0000000000abb474 in operator() (this=0x7f56b0cf8f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #32 operator() (__closure=0x7f56b0cf8f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #33 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #34 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #35 0x00007f56b4a6dea5 in start_thread (arg=0x7f5698bd1700) at pthread_create.c:307 #36 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 28 (LWP 44004): #0 0x0000000000583038 in construct (__p=, this=0x7f56993bfad0) at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #1 construct (__p=, __a=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/alloc_traits.h:512 #2 std::vector >::_M_realloc_insert(__gnu_cxx::__normal_iterator > >, char&&) (this=this@entry=0x7f56993bfad0, __position=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/vector.tcc:449 #3 0x0000000000583182 in emplace_back (this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:412 #4 push_back (__x=, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_vector.h:1204 #5 nlohmann::detail::lexer, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > >, nlohmann::detail::iterator_input_adapter<__gnu_cxx::__normal_iterator, std::allocator > > > >::get (this=0x7f56993bfa98) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/detail/input/lexer.hpp:1346 #6 0x0000000000584629 in skip_bom (this=0x7f56993bfa98) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/detail/input/lexer.hpp:1501 #7 nlohmann::detail::lexer, std::allocator >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector > >, nlohmann::detail::iterator_input_adapter<__gnu_cxx::__normal_iterator, std::allocator > > > >::scan (this=0x7f56993bfa98) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/detail/input/lexer.hpp:1504 #8 0x0000000000869485 in get_token (this=0x7f56993bfa70) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:322 #9 parser (skip_comments=false, allow_exceptions_=true, cb=..., adapter=, this=0x7f56993bfa70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/detail/input/parser.hpp:72 #10 parser > > > (ignore_comments=false, allow_exceptions=true, cb=..., adapter=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/json.hpp:204 #11 parse&> (ignore_comments=false, allow_exceptions=true, cb=..., i=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/json.exploded/include/nlohmann/json.hpp:6654 #12 vbucket_transition_state::fromItem(Item const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket_state.cc:34 #13 0x0000000000842406 in EPBucket::flushVBucket_UNLOCKED(LockedVBucketPtr) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:620 #14 0x00000000008432ef in EPBucket::flushVBucket(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:381 #15 0x00000000006ee2c6 in Flusher::flushVB (this=0x7f55cc3efc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:293 #16 0x00000000006eec80 in Flusher::step(GlobalTask*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:203 #17 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #18 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f56993c0650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #19 0x0000000000ac344e in operator() (this=0x7f56993c0650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #20 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #21 0x0000000000c215b0 in operator() (this=0x7f56993c0840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #22 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b11df500, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #23 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11df500, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #24 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #25 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #26 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #27 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #28 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #29 0x0000000000abb474 in operator() (this=0x7f56b0cf8f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #30 operator() (__closure=0x7f56b0cf8f00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #31 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #32 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #33 0x00007f56b4a6dea5 in start_thread (arg=0x7f56993d2700) at pthread_create.c:307 #34 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 27 (LWP 44003): #0 CheckpointManager::getMemOverheadAllocatorBytesIndex (this=0x7f55c4a45a40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/checkpoint_manager.cc:1646 #1 0x00000000007362dc in VBucket::getCMMemOverheadAllocatorBytesIndex (this=this@entry=0x7f55c4a6e500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.cc:316 #2 0x0000000000733a2b in VBucketCountVisitor::visitBucket (this=0x7f557442fe80, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:45 #3 0x0000000000733ca0 in VBucketStatAggregator::visitBucket (this=, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:112 #4 0x00000000007034da in KVBucket::visit(VBucketVisitor&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2383 #5 0x000000000071428c in KVBucket::getAggregatedVBucketStats(BucketStatCollector const&, cb::prometheus::Cardinality) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1333 #6 0x00000000006d2700 in EventuallyPersistentEngine::doEngineStatsLowCardinality(BucketStatCollector const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:3190 #7 0x00000000006d492b in doEngineStats (collector=..., this=0x7f55c5200000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:5104 #8 EventuallyPersistentEngine::getStats(void const*, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, void const*)> const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:5104 #9 0x0000000000710dfa in KVBucket::snapshotStats(bool) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #10 0x00000000007326f2 in StatSnap::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.h:661 #11 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #12 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5699bc1650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #13 0x0000000000ac344e in operator() (this=0x7f5699bc1650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #14 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #15 0x0000000000c215b0 in operator() (this=0x7f5699bc1840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #16 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b11df500, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #17 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11df500, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #18 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #19 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #20 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #21 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #22 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #23 0x0000000000abb474 in operator() (this=0x7f56b0cf8ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #24 operator() (__closure=0x7f56b0cf8ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #25 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #26 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #27 0x00007f56b4a6dea5 in start_thread (arg=0x7f5699bd3700) at pthread_create.c:307 #28 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 26 (LWP 44002): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2587633176, absSteadyTime=absSteadyTime@entry=0x7f569a3c25f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbda568) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbda568, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569a3c2618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbda568) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbda568, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbda568, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b0cf8dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b0cf8dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f569a3d4700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 25 (LWP 44001): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2596025880, absSteadyTime=absSteadyTime@entry=0x7f569abc35f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdc320) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdc320, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569abc3618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdc320) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdc320, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11c5d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdc320, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000abb474 in operator() (this=0x7f56b0cf8d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #18 operator() (__closure=0x7f56b0cf8d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f56b4a6dea5 in start_thread (arg=0x7f569abd5700) at pthread_create.c:307 #22 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 24 (LWP 44000): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2604418584, absSteadyTime=absSteadyTime@entry=0x7f569b3c45f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdad90) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdad90, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569b3c4618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdad90) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdad90, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11c5d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdad90, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000abb474 in operator() (this=0x7f56b0cf8c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #18 operator() (__closure=0x7f56b0cf8c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f56b4a6dea5 in start_thread (arg=0x7f569b3d6700) at pthread_create.c:307 #22 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 23 (LWP 43999): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2612811288, absSteadyTime=absSteadyTime@entry=0x7f569bbc55f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdc410) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdc410, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569bbc5618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdc410) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdc410, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdc410, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b0cf8d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b0cf8d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f569bbd7700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 22 (LWP 43998): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2646521368, absSteadyTime=absSteadyTime@entry=0x7f569dbeb5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbd9e90) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbd9e90, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569dbeb618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbd9e90) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbd9e90, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11c5d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbd9e90, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000abb474 in operator() (this=0x7f56b0cf8cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #18 operator() (__closure=0x7f56b0cf8cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f56b4a6dea5 in start_thread (arg=0x7f569dbfd700) at pthread_create.c:307 #22 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 21 (LWP 43997): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2654914072, absSteadyTime=absSteadyTime@entry=0x7f569e3ec5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdbea0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdbea0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569e3ec618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdbea0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdbea0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdbea0, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b0cf8c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b0cf8c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f569e3fe700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 20 (LWP 43996): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2663306776, absSteadyTime=absSteadyTime@entry=0x7f569ebed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbdea20) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbdea20, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f569ebed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbdea20) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbdea20, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11d97 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > ( timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbdea20, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000abb474 in operator() (this=0x7f56b0cf8c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 operator() (__closure=0x7f56b0cf8c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007f56b4a6dea5 in start_thread (arg=0x7f569ebff700) at pthread_create.c:307 #23 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 19 (LWP 43995): #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x0000000000c03d6d in nativeFutexWaitImpl (waitMask=waitMask@entry=2763970072, absSteadyTime=absSteadyTime@entry=0x7f56a4bed5f8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7f569bbd9f98) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7f569bbd9f98, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7f56a4bed618, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000c0d62a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7f569bbd9f98) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > (futex=futex@entry=0x7f569bbd9f98, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000c11c5d in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > (this=this@entry=0x7f569bbd9f98, deadline=..., opt=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000c122c6 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x2cf0f40, deadline=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000c14d80 in try_wait_for > (timeout=, this=0x2cf0f40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x2cf0f00, time=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000c0bfd7 in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b11de900, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000abb474 in operator() (this=0x7f56b0cf8b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #18 operator() (__closure=0x7f56b0cf8b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a4bff700) at pthread_create.c:307 #22 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 18 (LWP 43994): #0 0x00007f56b23b60e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f56b3d493b5 in epoll_dispatch (base=0x7f56b0c8cb00, tv=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f56b3d40366 in event_base_loop (base=0x7f56b0c8cb00, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e88930, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e88930) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e88930) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e88920) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #7 0x0000000000baf779 in run (this=0x7f56b0cf0840) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a57fa700) at pthread_create.c:307 #10 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 17 (LWP 43993): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f56b4a6fe9b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007f56b4a6fd68 in __GI___pthread_mutex_lock (mutex=mutex@entry=0x7f55c0408098) at ../nptl/pthread_mutex_lock.c:78 #3 0x00007f56b3fbe599 in malloc_mutex_lock_final (mutex=0x7f55c0408058) at include/jemalloc/internal/mutex.h:155 #4 je_malloc_mutex_lock_slow (mutex=mutex@entry=0x7f55c0408058) at src/mutex.c:85 #5 0x00007f56b3f7ba1d in malloc_mutex_lock (mutex=, tsdn=0x7f56a5fea220) at include/jemalloc/internal/mutex.h:221 #6 je_arena_bin_choose_lock (binshard=, binind=4, arena=0x7f55c04008c0, tsdn=0x7f56a5fea220) at src/arena.c:1375 #7 je_arena_tcache_fill_small (tsdn=0x7f56a5fea220, arena=0x7f55c04008c0, tcache=0x7f55c986e000, tbin=0x7f55c986e070, binind=4, prof_accumbytes=) at src/arena.c:1392 #8 0x00007f56b3feaeb5 in je_tcache_alloc_small_hard (tsdn=tsdn@entry=0x7f56a5fea220, arena=, tcache=tcache@entry=0x7f55c986e000, tbin=tbin@entry=0x7f55c986e070, binind=, tcache_success=tcache_success@entry=0x7f56a5fe5740) at src/tcache.c:94 #9 0x00007f56b3f6f9cb in tcache_alloc_small (slow_path=false, zero=, binind=, size=64, tcache=0x7f55c986e000, arena=, tsd=) at include/jemalloc/internal/tcache_inlines.h:60 #10 arena_malloc (slow_path=false, tcache=0x7f55c986e000, zero=, ind=, size=64, arena=, tsdn=) at include/jemalloc/internal/arena_inlines_b.h:165 #11 iallocztm (slow_path=false, arena=, is_internal=false, tcache=0x7f55c986e000, zero=, ind=, size=64, tsdn=) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:53 #12 imalloc_no_sample (ind=, usize=64, size=64, tsd=0x7f56a5fea220, dopts=, sopts=) at src/jemalloc.c:1949 #13 imalloc_body (tsd=0x7f56a5fea220, dopts=, sopts=) at src/jemalloc.c:2123 #14 imalloc (dopts=, sopts=) at src/jemalloc.c:2258 #15 je_mallocx (size=size@entry=51, flags=) at src/jemalloc.c:3137 #16 0x0000000000bcb6dd in cb::_JEArenaMalloc::malloc (size=size@entry=51) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/je_arena_malloc.cc:266 #17 0x000000000056bb6f in malloc (size=51) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_malloc_arena.cc:39 #18 cb_malloc (size=51) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_malloc_arena.cc:39 #19 0x0000000000bbe0c9 in operator new(unsigned long) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/global_new_replacement.cc:63 #20 0x0000000000a72331 in std::__cxx11::basic_string, std::allocator >::_M_assign(std::__cxx11::basic_string, std::allocator > const&) () at /opt/gcc-10.2.0/include/c++/10.2.0/bitset:411 #21 0x00000000009f7362 in assign (__str=..., this=0x7f56a5fe5bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:188 #22 operator= (__str=..., this=0x7f56a5fe5bf0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/basic_string.h:667 #23 magma::AtomicLSMTreeStats::Set(magma::LSMTreeStats&, magma::TableStats*, magma::TableStats*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:188 #24 0x00000000009f7924 in magma::LSMTree::GetStats() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/stats.cc:102 #25 0x000000000099f64f in magma::KVStore::GetKVStoreStats() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/stats.cc:43 #26 0x000000000096ff2a in magma::Magma::GetKVStoreStats(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db_stats.cc:462 #27 0x00000000008b8416 in MagmaMemoryTrackingProxy::GetStatsForDbInfo(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore/magma-kvstore/magma-memory-tracking-proxy.cc:296 #28 0x0000000000897663 in MagmaKVStore::getDbFileInfo(Vbid) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #29 0x0000000000848336 in EPVBucket::getHistoryDiskSize() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucket.h:448 #30 0x0000000000733bde in VBucketCountVisitor::visitBucket (this=0x7f55bf99fa80, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:92 #31 0x0000000000733ca0 in VBucketStatAggregator::visitBucket (this=, vb=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vb_count_visitor.cc:112 #32 0x00000000007034da in KVBucket::visit(VBucketVisitor&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2383 #33 0x000000000071428c in KVBucket::getAggregatedVBucketStats(BucketStatCollector const&, cb::prometheus::Cardinality) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:1333 #34 0x00000000006d2700 in EventuallyPersistentEngine::doEngineStatsLowCardinality(BucketStatCollector const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:3190 #35 0x00000000006d492b in doEngineStats (collector=..., this=0x7f55c0200000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:5104 #36 EventuallyPersistentEngine::getStats(void const*, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, void const*)> const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:5104 #37 0x00000000006d5709 in EventuallyPersistentEngine::get_stats(CookieIface const&, std::basic_string_view >, std::basic_string_view >, std::function >, std::basic_string_view >, void const*)> const&) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #38 0x0000000000597768 in bucket_get_stats(Cookie&, std::basic_string_view >, cb::sized_buffer, std::function >, std::basic_string_view >, void const*)> const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:372 #39 0x00000000006581ea in stat_all_stats(std::__cxx11::basic_string, std::allocator > const&, Cookie&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:472 #40 0x000000000065912f in StatsCommandContext::doStats() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:758 #41 0x000000000065a760 in StatsCommandContext::step() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:668 #42 0x000000000065e891 in SteppableCommandContext::drive() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:36 #43 0x000000000060f66b in Cookie::doExecute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/mcbp/protocol/header.h:110 #44 0x000000000060f700 in Cookie::execute (this=this@entry=0x7f559eba2500, useStartTime=useStartTime@entry=true) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:168 #45 0x00000000005f0c23 in Connection::executeCommandPipeline() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.h:472 #46 0x00000000005f0ed2 in Connection::executeCommandsCallback() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:801 #47 0x00000000005f155b in Connection::rw_callback(bufferevent*, void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:950 #48 0x00007f56b3d36d7e in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7f55a85b4e80) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208 #49 0x00007f56b3d3fcf1 in event_process_active_single_queue (base=base@entry=0x7f56b0c8c800, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #50 0x00007f56b3d4053f in event_process_active (base=0x7f56b0c8c800) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #51 event_base_loop (base=0x7f56b0c8c800, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #52 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e87d00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #53 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e87d00) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #54 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e87d00) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #55 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e87cf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #56 0x0000000000baf779 in run (this=0x7f56b11b7550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #57 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #58 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a5ffb700) at pthread_create.c:307 #59 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 16 (LWP 43992): #0 0x00007f56b23b60e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f56b3d493b5 in epoll_dispatch (base=0x7f56b0c8c500, tv=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f56b3d40366 in event_base_loop (base=0x7f56b0c8c500, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e870d0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e870d0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e870d0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e870c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #7 0x0000000000baf779 in run (this=0x7f56b0cf0870) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a67fc700) at pthread_create.c:307 #10 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 15 (LWP 43991): #0 0x00007f56b23b60e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f56b3d493b5 in epoll_dispatch (base=0x7f56b0c8c200, tv=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f56b3d40366 in event_base_loop (base=0x7f56b0c8c200, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e864a0, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e864a0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e864a0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e86490) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #7 0x0000000000baf779 in run (this=0x7f56b0cf08a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a6ffd700) at pthread_create.c:307 #10 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 14 (LWP 43990): #0 __GI___pthread_mutex_lock (mutex=0x7f56b0dba970) at ../nptl/pthread_mutex_lock.c:62 #1 0x00007f56b3d366ac in bufferevent_incref_and_lock_ (bufev=bufev@entry=0x7f56b0df3200) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:683 #2 0x00007f56b3d36ac2 in bufferevent_trigger (bufev=0x7f56b0df3200, iotype=, options=65540) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:279 #3 0x00000000005e3c76 in Connection::triggerCallback (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:1240 #4 0x00000000005be729 in operator() (__closure=0x7f55bbb35af0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:347 #5 void folly::detail::function::FunctionTraits::callSmall(folly::detail::function::Data&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:387 #6 0x0000000000c3005a in operator() (this=0x7f55bbb35af0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #7 messageAvailable (msg=, this=0x7f56b11e5900) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:137 #8 folly::NotificationQueue >::Consumer::consumeMessages(bool, unsigned long*) (this=0x7f56b11e5900, isDrain=, numConsumed=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/NotificationQueue.h:729 #9 0x00007f56b3d3ff9a in event_persist_closure (ev=, base=0x7f56b0c8bf00) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1629 #10 event_process_active_single_queue (base=base@entry=0x7f56b0c8bf00, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1688 #11 0x00007f56b3d4053f in event_process_active (base=0x7f56b0c8bf00) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #12 event_base_loop (base=0x7f56b0c8bf00, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #13 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e85870, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #14 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e85870) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #15 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e85870) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #16 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e85860) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #17 0x0000000000baf779 in run (this=0x7f56b11b74f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #18 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #19 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a77fe700) at pthread_create.c:307 #20 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 13 (LWP 43989): #0 0x00007f56b23b60e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f56b3d493b5 in epoll_dispatch (base=0x7f56b0c8bc00, tv=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f56b3d40366 in event_base_loop (base=0x7f56b0c8bc00, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e84c40, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e84c40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e84c40) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e84c30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #7 0x0000000000baf779 in run (this=0x7f56b11b73d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f56b4a6dea5 in start_thread (arg=0x7f56a7fff700) at pthread_create.c:307 #10 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 12 (LWP 43988): #0 evbuffer_chain_new (size=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/buffer.c:181 #1 0x00007f56b3d32c2a in evbuffer_add (buf=0x7f5670e731e0, data_in=0x7f55c8c96000, datlen=2040) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/buffer.c:1796 #2 0x00007f56b3d36180 in bufferevent_write (bufev=, data=, size=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:454 #3 0x00000000005e3dc3 in Connection::copyToOutputStream(gsl::span > >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:1278 #4 0x00000000005ed456 in copyToOutputStream (data4=..., data3=..., data2=..., data1=..., this=0x7f5654561000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.h:404 #5 Connection::sendResponse(Cookie&, cb::mcbp::Status, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, std::unique_ptr >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:1823 #6 0x0000000000682377 in mcbp_response_handler(std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp.cc:86 #7 0x000000000068286a in __invoke_impl, std::basic_string_view, std::basic_string_view, unsigned char, cb::mcbp::Status, unsigned long, void const*), std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*> (__f=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:88 #8 __invoke_r, std::basic_string_view, std::basic_string_view, unsigned char, cb::mcbp::Status, unsigned long, void const*), std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*> (__fn=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:113 #9 std::_Function_handler >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*), bool (*)(std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*)>::_M_invoke(std::_Any_data const&, std::basic_string_view >&&, std::basic_string_view >&&, std::basic_string_view >&&, unsigned char&&, cb::mcbp::Status&&, unsigned long&&, void const*&&) (__functor=..., __args#0=, __args#1=, __args#2=, __args#3=, __args#4=, __args#5=, __args#6=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #10 0x00000000006d7299 in operator() (__args#6=, __args#5=, __args#4=, __args#3=, __args#2=..., __args#1=..., __args#0=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #11 __invoke_impl, std::basic_string_view, std::basic_string_view, unsigned char, cb::mcbp::Status, long unsigned int, void const*)>&, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, long unsigned int, void const*> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #12 __invoke, std::basic_string_view, std::basic_string_view, unsigned char, cb::mcbp::Status, long unsigned int, void const*)>&, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, long unsigned int, void const*> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95 #13 operator() >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, long unsigned int, void const*> ( this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/refwrap.h:349 #14 operator() >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, long unsigned int, void const*> ( this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:346 #15 __invoke_impl [with auto:53 = std::reference_wrapper, std::basic_string_view, std::basic_string_view, unsigned char, cb::mcbp::Status, long unsigned int, void const*)> >]::&, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, long unsigned int, void const*> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #16 __invoke_r [with auto:53 = std::reference_wrapper, std::basic_string_view, std::basic_string_view, unsigned char, cb::mcbp::Status, long unsigned int, void const*)> >]::&, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, long unsigned int, void const*> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:113 #17 _ZNSt17_Function_handlerIFbSt17basic_string_viewIcSt11char_traitsIcEES3_S3_hN2cb4mcbp6StatusEmPKvEZNK19makeExitBorderGuardMUlOT_E_clISt17reference_wrapperIKSt8functionIS9_EEEEDaSB_EUlDpOT_E_E9_M_invokeERKSt9_Any_dataOS3_SR_SR_OhOS6_OmOS8_ ( __functor=..., __args#0=, __args#1=, __args#2=, __args#3=, __args#4=, __args#5=, __args#6=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #18 0x00000000006b2327 in operator() (__args#6=, __args#5=, __args#4=, __args#3=, __args#2=..., __args#1=..., __args#0=..., this=this@entry=0x7f56ac8e3f80) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #19 sendResponse (response=..., key=..., ext=..., body=..., datatype=datatype@entry=0 '\000', status=cb::mcbp::Success, cas=0, cookie=0x7f5654561280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:152 #20 0x00000000006caa61 in EventuallyPersistentEngine::getAllVBucketSequenceNumbers(CookieIface const*, cb::mcbp::Request const&, std::function >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*)> const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:6976 #21 0x00000000006d1a6d in EventuallyPersistentEngine::processUnknownCommandInner(CookieIface const*, cb::mcbp::Request const&, std::function >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*)> const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:1175 #22 0x00000000006d1b7a in EventuallyPersistentEngine::unknown_command(CookieIface const*, cb::mcbp::Request const&, std::function >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*)> const&) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #23 0x0000000000594b26 in bucket_unknown_command(Cookie&, std::function >, std::basic_string_view >, std::basic_string_view >, unsigned char, cb::mcbp::Status, unsigned long, void const*)> const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:28 #24 0x000000000061a585 in process_bin_unknown_packet (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:108 #25 0x000000000060f66b in Cookie::doExecute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/mcbp/protocol/header.h:110 #26 0x000000000060f700 in Cookie::execute (this=this@entry=0x7f5654561280, useStartTime=useStartTime@entry=true) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:168 #27 0x00000000005f0c23 in Connection::executeCommandPipeline() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.h:472 #28 0x00000000005f0ed2 in Connection::executeCommandsCallback() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:801 #29 0x00000000005f155b in Connection::rw_callback(bufferevent*, void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:950 #30 0x00007f56b3d36d7e in bufferevent_run_deferred_callbacks_unlocked (cb=, arg=0x7f5654561500) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208 #31 0x00007f56b3d3fcf1 in event_process_active_single_queue (base=base@entry=0x7f56b0c8b900, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726 #32 0x00007f56b3d4053f in event_process_active (base=0x7f56b0c8b900) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789 #33 event_base_loop (base=0x7f56b0c8b900, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012 #34 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b0e84010, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #35 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b0e84010) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #36 0x0000000000c2cb26 in folly::EventBase::loopForever (this=this@entry=0x7f56b0e84010) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #37 0x00000000005bd749 in worker_libevent (arg=0x7f56b0e84000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257 #38 0x0000000000baf779 in run (this=0x7f56b11b72b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #39 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #40 0x00007f56b4a6dea5 in start_thread (arg=0x7f56ac8f6700) at pthread_create.c:307 #41 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 11 (LWP 43987): #0 0x00007f56b23aaddd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000aecb38 in master_thread_run (ctx=0x2ceba00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 master_thread (thread_func_param=0x2ceba00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19386 #3 0x00007f56b4a6dea5 in start_thread (arg=0x7f56af0fb700) at pthread_create.c:307 #4 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 10 (LWP 43986): #0 0x00000000008e0329 in _M_add_ref_lock_nothrow (this=0x7f55a181e000) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr_base.h:298 #1 __shared_count (__r=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr_base.h:914 #2 __shared_ptr (__r=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr_base.h:1382 #3 shared_ptr (__r=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr.h:417 #4 lock (this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr.h:749 #5 lock (this=0x7f55a12c5880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/cursor.h:76 #6 ActiveStream::getItemsRemaining() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:2287 #7 0x00000000007ff681 in operator() (vt=..., __closure=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:1987 #8 for_each > > >::aha_iterator > > >, std::pair > > > >, DcpProducer::getStreamAggStats() const:: > (__f=..., __last=..., __first=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_algo.h:3839 #9 DcpProducer::getStreamAggStats() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:1994 #10 0x00000000007ff80c in DcpProducer::aggregateQueueStats (this=, aggregator=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:1618 #11 0x00000000006df030 in aggregate (tc=0x7f56b0da9080, conn=..., this=0x7f56ae8e4130) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:4116 #12 operator() (tc=..., this=0x7f56ae8e4130) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:4133 #13 DcpConnMap::each (this=, f=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/dcpconnmap_impl.h:32 #14 0x00000000006d57cc in EventuallyPersistentEngine::doConnAggStatsInner(BucketStatCollector const&, std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:4259 #15 0x00000000006d620f in EventuallyPersistentEngine::get_prometheus_stats(BucketStatCollector const&, cb::prometheus::Cardinality) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:357 #16 0x00000000005b7e1d in operator() (bucket=..., __closure=0x7f56ae8e4390) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stats.cc:249 #17 __invoke_impl&, Bucket&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #18 __invoke_r&, Bucket&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:113 #19 std::_Function_handler::_M_invoke(std::_Any_data const&, Bucket&) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #20 0x00000000005d0294 in operator() (__args#0=..., this=0x7f56ae8e4390) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #21 BucketManager::forEach(std::function) (this=, fn=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/buckets.cc:365 #22 0x00000000005b6fac in server_prometheus_stats(StatCollector const&, cb::prometheus::Cardinality) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #23 0x0000000000ae332a in operator() (__args#1=, __args#0=..., this=0x7f56b0cf82a0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #24 cb::prometheus::MetricServer::KVCollectable::Collect (this=0x7f56b0cf8290) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/statistics/prometheus.cc:122 #25 0x0000000000afb62b in prometheus::detail::CollectMetrics (collectables=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/pull/src/metrics_collector.cc:18 #26 0x0000000000afa668 in prometheus::detail::MetricsHandler::handleGet (this=0x7f56b11e4800, conn=0x2cea2a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/pull/src/handler.cc:129 #27 0x0000000000ae8807 in CivetServer::requestHandler (conn=0x2cea2a8, cbdata=0x7f56b11e4800) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/CivetServer.cpp:152 #28 0x0000000000af55ad in handle_request (conn=conn@entry=0x2cea2a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:14657 #29 0x0000000000af6609 in process_new_connection (conn=0x2cea2a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18673 #30 worker_thread_run (conn=0x2cea2a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19090 #31 worker_thread (thread_func_param=0x2cea2a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #32 0x00007f56b4a6dea5 in start_thread (arg=0x7f56ae8fa700) at pthread_create.c:307 #33 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 9 (LWP 43985): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x0000000000af5bcb in consume_socket (thread_index=, sp=0x2cea1b0, ctx=0x2ceba00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x2ce98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x2ce98c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007f56b4a6dea5 in start_thread (arg=0x7f56ae0f9700) at pthread_create.c:307 #5 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 8 (LWP 43982): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000068c207 in __gthread_cond_timedwait (__abs_timeout=0x7f56ad0e59b0, __mutex=, __cond=0x7f56b0d193f8) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7f56b0d193f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7f56b0d193f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7f56b0d193f8) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 AuditImpl::consume_events() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/audit.cc:414 #6 0x0000000000baf779 in run (this=0x7f56b11b6fb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #7 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #8 0x00007f56b4a6dea5 in start_thread (arg=0x7f56ad0f7700) at pthread_create.c:307 #9 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 7 (LWP 43981): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000005ffdf6 in __gthread_cond_timedwait (__abs_timeout=0x7f56ad8e6980, __mutex=0x7f56b11e6350, __cond=0x7f56b11e6378) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=, __lock=, __atime=, this=0x7f56b11e6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=, this=0x7f56b11e6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=, __lock=, this=0x7f56b11e6378) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 ExternalAuthManagerThread::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/external_auth_manager_thread.cc:91 #6 0x0000000000bc7473 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #7 0x0000000000baf779 in run (this=0x7f56b11b7010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007f56b4a6dea5 in start_thread (arg=0x7f56ad8f8700) at pthread_create.c:307 #10 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 6 (LWP 43977): #0 0x00007f56b23aaddd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000005b8451 in get_command (buffsize=80, buffer=0x7f56af8ea990 "@\206\n\265V\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000baf779 in run (this=0x7f56b11b6c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007f56b4a6dea5 in start_thread (arg=0x7f56af8fc700) at pthread_create.c:307 #6 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 5 (LWP 43976): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000b7666a in __gthread_cond_timedwait (__abs_timeout=0x7f56b00eb9b0, __mutex=, __cond=0x7f56b11b82d8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #2 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f56b11b82d8) at /usr/local/include/c++/7.3.0/condition_variable:166 #3 wait_until > > (__atime=..., __lock=..., this=0x7f56b11b82d8) at /usr/local/include/c++/7.3.0/condition_variable:106 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=0x7f56b11b82d8) at /usr/local/include/c++/7.3.0/condition_variable:129 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=0x7f56b11b82d8) at /usr/local/include/c++/7.3.0/condition_variable:145 #6 operator() (__closure=0x7f56b11c2ac8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/periodic_worker-inl.h:25 #7 __invoke_impl&, std::chrono::seconds):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7f56b11c2ac8) at /usr/local/include/c++/7.3.0/thread:234 #10 operator() (this=0x7f56b11c2ac8) at /usr/local/include/c++/7.3.0/thread:243 #11 std::thread::_State_impl&, std::chrono::seconds):: > > >::_M_run(void) (this=0x7f56b11c2ac0) at /usr/local/include/c++/7.3.0/thread:186 #12 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007f56b4a6dea5 in start_thread (arg=0x7f56b00fd700) at pthread_create.c:307 #14 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 4 (LWP 43975): #0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x00007f56b4a76d3c in _L_cond_lock_847 () from /lib64/libpthread.so.0 #2 0x00007f56b4a76bd1 in __pthread_mutex_cond_lock (mutex=0x7f56b11e5010) at ../nptl/pthread_mutex_lock.c:78 #3 0x00007f56b4a71eb4 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:341 #4 0x0000000000b768aa in __gthread_cond_timedwait (__abs_timeout=0x7f56b08ec800, __mutex=, __cond=0x7f56b11e5038) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #5 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f56b11e5038) at /usr/local/include/c++/7.3.0/condition_variable:166 #6 wait_until > > (__atime=..., __lock=..., this=0x7f56b11e5038) at /usr/local/include/c++/7.3.0/condition_variable:106 #7 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > ( __p=..., __atime=..., __lock=..., this=0x7f56b11e5038) at /usr/local/include/c++/7.3.0/condition_variable:129 #8 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7f56b11e5038) at /usr/local/include/c++/7.3.0/condition_variable:145 #9 dequeue_for (wait_duration=..., popped_item=..., this=0x7f56b11e5010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/mpmc_blocking_q.h:58 #10 spdlog::details::thread_pool::process_next_msg_ (this=this@entry=0x7f56b11e5010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:99 #11 0x0000000000b76bf8 in worker_loop_ (this=0x7f56b11e5010) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:90 #12 operator() (__closure=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/spdlog/spdlog-prefix/src/spdlog/include/spdlog/details/thread_pool-inl.h:28 #13 __invoke_impl):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #14 __invoke):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 _M_invoke<0> (this=) at /usr/local/include/c++/7.3.0/thread:234 #16 operator() (this=) at /usr/local/include/c++/7.3.0/thread:243 #17 std::thread::_State_impl):: > > >::_M_run(void) (this=) at /usr/local/include/c++/7.3.0/thread:186 #18 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #19 0x00007f56b4a6dea5 in start_thread (arg=0x7f56b08fe700) at pthread_create.c:307 #20 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 3 (LWP 43970): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00007f56b3f82567 in background_thread_sleep (tsdn=, interval=7525000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7f56b19ee220) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7f56b19ee220) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007f56b4a6dea5 in start_thread (arg=0x7f56b19ff700) at pthread_create.c:307 #7 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 Thread 2 (LWP 43964): #0 0x00007f56b23b60e3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f56b3d493b5 in epoll_dispatch (base=0x7f56b0c8b300, tv=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007f56b3d40366 in event_base_loop (base=0x7f56b0c8b300, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000c2ac97 in folly::EventBase::loopBody (this=this@entry=0x7f56b11e6500, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000c2b166 in folly::EventBase::loop (this=this@entry=0x7f56b11e6500) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000c2cb26 in folly::EventBase::loopForever (this=0x7f56b11e6500) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000573a7c in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1063 #7 0x00007f56b22d9555 in __libc_start_main (main=0x546e20 , argc=3, argv=0x7ffef9047818, init=, fini=, rtld_fini=, stack_end=0x7ffef9047808) at ../csu/libc-start.c:266 #8 0x000000000056baa7 in _start () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:322 Thread 1 (LWP 44026): #0 0x00007f56b22ed387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007f56b22eea78 in __GI_abort () at abort.c:90 #2 0x00007f56b2c3863c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 #3 0x0000000000b53a8b in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:88 #4 0x00007f56b2c438f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 #5 0x00007f56b2c43961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 #6 0x00007f56b2c43bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f5604000980, tinfo=tinfo@entry=0x109a3a8 >, dest=dest@entry=0xbb69f0 ::~error_info_injector()>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 #7 0x0000000000542ccd in cb::throwWithTrace (exception=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/boost.exploded/include/boost/exception/info.hpp:129 #8 0x00000000008db6ac in ThrowExceptionPolicy::nonMonotonic(unsigned long const&, unsigned long const&) [clone .isra.0] () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/char_traits.h:322 #9 0x00000000008ddf5d in store (memoryOrder=std::memory_order_seq_cst, desired=, this=0x7f55bf847ec0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1525 #10 ActiveStream::snapshot(ActiveStream::OutstandingItemsResult const&, std::deque >, std::allocator > > >&, unsigned long, std::optional) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1484 #11 0x00000000008e6be1 in ActiveStream::processItems(ActiveStream::OutstandingItemsResult&, std::lock_guard const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1336 #12 0x00000000008e743c in ActiveStream::nextCheckpointItemTask(std::lock_guard const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1009 #13 0x00000000008e7564 in ActiveStream::nextCheckpointItemTask() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:993 #14 0x00000000008ee299 in ActiveStreamCheckpointProcessorTask::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream_checkpoint_processor_task.cc:52 #15 0x0000000000ac2099 in GlobalTask::execute(std::basic_string_view >) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/globaltask.cc:98 #16 0x0000000000abb77a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f5613fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:309 #17 0x0000000000ac344e in operator() (this=0x7f5613fed650) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #18 CancellableCPUExecutor::add(GlobalTask*, folly::Function)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/cancellable_cpu_executor.cc:42 #19 0x0000000000c215b0 in operator() (this=0x7f5613fed840) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #20 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f56b0d46d00, thread=..., task=task@entry=) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #21 0x0000000000c0c04a in folly::CPUThreadPoolExecutor::threadRun (this=0x7f56b0d46d00, thread=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #22 0x0000000000c24569 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #23 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #24 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #25 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #26 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #27 0x0000000000abb474 in operator() (this=0x7f56b11f2240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #28 operator() (__closure=0x7f56b11f2240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/executor/folly_executorpool.cc:49 #29 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #30 0x00007f56b2c6cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #31 0x00007f56b4a6dea5 in start_thread (arg=0x7f5613fff700) at pthread_create.c:307 #32 0x00007f56b23b5b0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111