Basic crash dump analysis of cbcollect_info_ns_1@172.23.106.10_20210719-072921/a.core. Please send the file to support@couchbase.com -------------------------------------------------------------------------------- File information: -rwxr-xr-x. 1 couchbase couchbase 2140096 Jul 15 16:18 /opt/couchbase/bin/memcached c2117aa1d9a4a83fedbed2d626c4009e /opt/couchbase/bin/memcached -rw-r--r--. 1 root root 380928 Jul 19 03:01 cbcollect_info_ns_1@172.23.106.10_20210719-072921/a.core 6332e69d705136d6e2828cd3dcca4f93 cbcollect_info_ns_1@172.23.106.10_20210719-072921/a.core -------------------------------------------------------------------------------- Core file callstacks: GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /opt/couchbase/bin/memcached...Reading symbols from /usr/lib/debug/opt/couchbase/bin/memcached.debug...done. done. [New LWP 91686] [New LWP 91377] [New LWP 91380] [New LWP 91382] [New LWP 91383] [New LWP 91384] [New LWP 91385] [New LWP 91386] [New LWP 91387] [New LWP 91388] [New LWP 91389] [New LWP 91390] [New LWP 91391] [New LWP 91392] [New LWP 91393] [New LWP 91394] [New LWP 91674] [New LWP 91679] [New LWP 91680] [New LWP 91681] [New LWP 91682] [New LWP 91683] [New LWP 91684] [New LWP 91685] [New LWP 91687] [New LWP 91688] [New LWP 91689] warning: .dynamic section for "/usr/lib64/libm-2.17.so" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/lib64/ld-2.17.so" is not at the expected address (wrong library or version mismatch?) warning: .dynamic section for "/usr/lib64/librt-2.17.so" is not at the expected address (wrong library or version mismatch?) Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /usr/lib64/libthread_db-1.0.so line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error warning: File "/usr/lib64/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Failed to read a valid object file image from memory. Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f2790636f45 in __memcmp_sse4_1 () from /usr/lib64/libc-2.17.so Thread 27 (LWP 91689): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180a78 in ?? () #2 0x00007f2756ffb3e0 in ?? () #3 0x00007f278f1a26a8 in ?? () #4 0x00007f278f180a80 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f2756ffb380, __mutex=, __cond=0x7f278f1a26a8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2680, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2680, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=89 'Y') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=89 'Y') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180a40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06da20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06da20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 26 (LWP 91688): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180998 in ?? () #2 0x00007f27577fc3e0 in ?? () #3 0x00007f278f1a26a8 in ?? () #4 0x00007f278f1809a0 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f27577fc380, __mutex=, __cond=0x7f278f1a26a8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2680, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2680) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2680, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=62 '>') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=62 '>') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180960) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06da50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06da50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 25 (LWP 91687): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f1808b8 in ?? () #2 0x00007f2757ffd3e0 in ?? () #3 0x00007f278f1a2528 in ?? () #4 0x00007f278f1808c0 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f2757ffd380, __mutex=, __cond=0x7f278f1a2528) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2500, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2500, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=74 'J') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=74 'J') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06da80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06da80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 24 (LWP 91685): #0 0x00007f27905c6b7d in tsearch () from /usr/lib64/libc-2.17.so #1 0x00007f278b860b2a in write_entire_buffer (file=0x7f27859a3c80, buf=, nbytes=0, offset=139807719375408) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_write.cc:35 #2 0x00007f27856f5b00 in ?? () #3 0x00007f27859a3ca0 in ?? () #4 0x00007f278b868b70 in ?? () at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:498 from /opt/couchbase/lib/libcouchstore.so #5 0x000f80d61cd37ba6 in ?? () #6 0x00007f27856e6d40 in ?? () #7 0x00007f278ef7c000 in ?? () #8 0x00007f278b86850d in flush_buffer (buf=0x7f27859a3c80, errinfo=0x7f275d7f7aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:336 #9 BufferedFileOps::sync (this=, errinfo=0x7f275d7f7aa0, handle=0x905a) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:641 #10 0x00007f278b855d57 in precommit (db=db@entry=0x7f27859a3c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_db.cc:259 #11 0x00007f278b85a3dc in couchstore_commit (db=0x7f27859a3c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_db.cc:270 #12 0x00007f278bc8b64d in CouchKVStore::setVBucketState (this=this@entry=0x7f2785734000, vbucketId=..., vbstate=..., options=options@entry=VBSTATE_PERSIST_WITH_COMMIT) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1360 #13 0x00007f278bc8bc55 in CouchKVStore::snapshotVBucket (this=0x7f2785734000, vbucketId=..., vbstate=..., options=VBSTATE_PERSIST_WITH_COMMIT) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1412 #14 0x00007f278bb7bea6 in EPBucket::flushVBucket (this=0x7f278ef7c000, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:680 #15 0x00007f278bbcef7c in Flusher::flushVB (this=this@entry=0x7f27858e6000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:306 #16 0x00007f278bbcfeb9 in Flusher::step (this=0x7f27858e6000, task=0x7f278f174a90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:207 #17 0x00007f278bbc9854 in ExecutorThread::run (this=0x7f278f1806c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:190 #18 0x00007f279308df17 in run (this=0x7f278f06dae0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #19 platform_thread_wrap (arg=0x7f278f06dae0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #20 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #21 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #22 0x0000000000000000 in ?? () Thread 23 (LWP 91684): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180618 in ?? () #2 0x00007f275dff93e0 in ?? () #3 0x00007f278f1a2228 in ?? () #4 0x00007f278f180620 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f275dff9380, __mutex=, __cond=0x7f278f1a2228) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2200, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2200, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=135 '\207') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=135 '\207') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f1805e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06db10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06db10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 22 (LWP 91683): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180538 in ?? () #2 0x00007f275e7fa3e0 in ?? () #3 0x00007f278f1a2228 in ?? () #4 0x00007f278f180540 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f275e7fa380, __mutex=, __cond=0x7f278f1a2228) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2200, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2200, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=118 'v') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=118 'v') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06db40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06db40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 21 (LWP 91682): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180458 in ?? () #2 0x00007f275effb3e0 in ?? () #3 0x00007f278f1a23a8 in ?? () #4 0x00007f278f180460 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f275effb380, __mutex=, __cond=0x7f278f1a23a8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2380, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2380, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=4 '\004') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=4 '\004') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180420) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06db70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06db70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 20 (LWP 91681): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180378 in ?? () #2 0x00007f275f7fc3e0 in ?? () #3 0x00007f278f1a23a8 in ?? () #4 0x00007f278f180380 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f275f7fc380, __mutex=, __cond=0x7f278f1a23a8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2380, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2380, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=4 '\004') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=4 '\004') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06dba0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06dba0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 19 (LWP 91680): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f180298 in ?? () #2 0x00007f275fffd3e0 in ?? () #3 0x00007f278f1a23a8 in ?? () #4 0x00007f278f1802a0 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f275fffd380, __mutex=, __cond=0x7f278f1a23a8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2380, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2380, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=4 '\004') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=4 '\004') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f180260) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06dbd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06dbd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 18 (LWP 91679): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278f1800d8 in ?? () #2 0x00007f2784e7b3e0 in ?? () #3 0x00007f278f1a23a8 in ?? () #4 0x00007f278f1800e0 in ?? () #5 0x00007f278bc27d6b in __gthread_cond_timedwait (__abs_timeout=0x7f2784e7b380, __mutex=, __cond=0x7f278f1a23a8) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 wait_for (nanoSecs=..., lock=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/syncobject.h:53 #10 TaskQueue::_doSleep (this=this@entry=0x7f278f1a2380, t=..., lock=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:103 #11 0x00007f278bc294db in _sleepThenFetchNextTask (t=..., this=0x7f278f1a2380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:124 #12 TaskQueue::sleepThenFetchNextTask (this=this@entry=0x7f278f1a2380, thread=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/taskqueue.cc:167 #13 0x00007f278bbc00c7 in ExecutorPool::_nextTask (this=this@entry=0x7f278f093a00, t=..., tick=tick@entry=31 '\037') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:190 #14 0x00007f278bbc016d in ExecutorPool::nextTask (this=0x7f278f093a00, t=..., tick=tick@entry=31 '\037') at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorpool.cc:205 #15 0x00007f278bbc966f in ExecutorThread::run (this=0x7f278f1800a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:124 #16 0x00007f279308df17 in run (this=0x7f278f06dc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06dc00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () Thread 17 (LWP 91674): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000000000419560 in std::invalid_argument::~invalid_argument () #2 0x00007f278f076b00 in ?? () #3 0x00007f278f076b78 in ?? () #4 0x00007f278567c5b0 in ?? () #5 0x00007f278bc044ef in __gthread_cond_timedwait (__abs_timeout=0x3d, __mutex=, __cond=0xa) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=, this=0xa) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=, this=0xa) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_until >, MemoryTracker::statsThreadMainLoop(void*):: > (__atime=..., __lock=, this=0xa, __p=...) at /usr/local/include/c++/7.3.0/condition_variable:129 #9 wait_for, MemoryTracker::statsThreadMainLoop(void*):: > (__rtime=..., __lock=, this=0xa, __p=...) at /usr/local/include/c++/7.3.0/condition_variable:145 #10 MemoryTracker::statsThreadMainLoop (arg=0xffffffffffffff92) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/memory_tracker.cc:38 #11 0x00007f279308df17 in run (this=0x7f278fe1ccb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7f278fe1ccb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #14 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #15 0x0000000000000000 in ?? () Thread 16 (LWP 91394): #0 0x00007f27908a56d5 in pthread_cond_init@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000002700000000 in ?? () #2 0x00007f278f0ed698 in ?? () #3 0x00007f278f0ed5d0 in ?? () #4 0x0000000000000027 in ?? () #5 0x00007f275c41a430 in ?? () #6 0x00007f27910831bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f278f0ed698) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #7 std::condition_variable::wait (this=this@entry=0x7f278f0ed698, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #8 0x00000000004a2c31 in Executor::run (this=0x7f278f0ed540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #9 0x00007f27930997bf in Couchbase::Thread::thread_entry (this=0x7f278f0ed540) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #10 0x00007f279308df17 in run (this=0x7f278f06fd60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #11 platform_thread_wrap (arg=0x7f278f06fd60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #12 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #13 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #14 0x0000000000000000 in ?? () Thread 15 (LWP 91393): #0 0x00007f27908a56d5 in pthread_cond_init@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000002700000000 in ?? () #2 0x00007f278f0ed4d8 in ?? () #3 0x00007f278f0ed410 in ?? () #4 0x0000000000000027 in ?? () #5 0x00007f275c41a250 in ?? () #6 0x00007f27910831bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f278f0ed4d8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #7 std::condition_variable::wait (this=this@entry=0x7f278f0ed4d8, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #8 0x00000000004a2c31 in Executor::run (this=0x7f278f0ed380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #9 0x00007f27930997bf in Couchbase::Thread::thread_entry (this=0x7f278f0ed380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #10 0x00007f279308df17 in run (this=0x7f278f06faf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #11 platform_thread_wrap (arg=0x7f278f06faf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #12 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #13 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #14 0x0000000000000000 in ?? () Thread 14 (LWP 91392): #0 0x00007f27908a56d5 in pthread_cond_init@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000002800000000 in ?? () #2 0x00007f278f0ed318 in ?? () #3 0x00007f278f0ed250 in ?? () #4 0x0000000000000028 in ?? () #5 0x00007f278440c790 in ?? () #6 0x00007f27910831bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f278f0ed318) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #7 std::condition_variable::wait (this=this@entry=0x7f278f0ed318, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #8 0x00000000004a2c31 in Executor::run (this=0x7f278f0ed1c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #9 0x00007f27930997bf in Couchbase::Thread::thread_entry (this=0x7f278f0ed1c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #10 0x00007f279308df17 in run (this=0x7f278f06fdc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #11 platform_thread_wrap (arg=0x7f278f06fdc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #12 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #13 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #14 0x0000000000000000 in ?? () Thread 13 (LWP 91391): #0 0x00007f27908a56d5 in pthread_cond_init@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000002700000000 in ?? () #2 0x00007f278f0ed158 in ?? () #3 0x00007f278f0ed090 in ?? () #4 0x0000000000000027 in ?? () #5 0x00007f2755b83c70 in ?? () #6 0x00007f27910831bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7f278f0ed158) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:864 #7 std::condition_variable::wait (this=this@entry=0x7f278f0ed158, __lock=...) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #8 0x00000000004a2c31 in Executor::run (this=0x7f278f0ed000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:44 #9 0x00007f27930997bf in Couchbase::Thread::thread_entry (this=0x7f278f0ed000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #10 0x00007f279308df17 in run (this=0x7f278f06fd90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #11 platform_thread_wrap (arg=0x7f278f06fd90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #12 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #13 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #14 0x0000000000000000 in ?? () Thread 12 (LWP 91390): #0 0x00007f27905c9999 in fsetxattr () from /usr/lib64/libc-2.17.so #1 0x00007f279216c31a in epoll_wait (epfd=, events=, maxevents=, timeout=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll_sub.c:64 #2 0x00007f279216c7b5 in epoll_dispatch (base=0x281e9a0, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:462 #3 0x00007f27921644d2 in event_base_loop (base=0x281e9a0, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1947 #4 0x000000000046b359 in worker_libevent (arg=0x7f278f057028) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:218 #5 0x00007f279308df17 in run (this=0x7f278f06fb20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7f278f06fb20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #8 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #9 0x0000000000000000 in ?? () Thread 11 (LWP 91389): #0 0x00007f27905c9999 in fsetxattr () from /usr/lib64/libc-2.17.so #1 0x00007f279216c31a in epoll_wait (epfd=, events=, maxevents=, timeout=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll_sub.c:64 #2 0x00007f279216c7b5 in epoll_dispatch (base=0x281e270, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:462 #3 0x00007f27921644d2 in event_base_loop (base=0x281e270, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1947 #4 0x000000000046b359 in worker_libevent (arg=0x7f278f056d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:218 #5 0x00007f279308df17 in run (this=0x7f278f06fd30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7f278f06fd30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #8 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #9 0x0000000000000000 in ?? () Thread 10 (LWP 91388): #0 0x00007f27905c9999 in fsetxattr () from /usr/lib64/libc-2.17.so #1 0x00007f279216c31a in epoll_wait (epfd=, events=, maxevents=, timeout=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll_sub.c:64 #2 0x00007f279216c7b5 in epoll_dispatch (base=0x281dc40, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:462 #3 0x00007f27921644d2 in event_base_loop (base=0x281dc40, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1947 #4 0x000000000046b359 in worker_libevent (arg=0x7f278f056ab8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:218 #5 0x00007f279308df17 in run (this=0x7f278f0834e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7f278f0834e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #8 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #9 0x0000000000000000 in ?? () Thread 9 (LWP 91387): #0 0x00007f27905c9999 in fsetxattr () from /usr/lib64/libc-2.17.so #1 0x00007f279216c31a in epoll_wait (epfd=, events=, maxevents=, timeout=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll_sub.c:64 #2 0x00007f279216c7b5 in epoll_dispatch (base=0x281d650, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:462 #3 0x00007f27921644d2 in event_base_loop (base=0x281d650, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1947 #4 0x000000000046b359 in worker_libevent (arg=0x7f278f056800) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:218 #5 0x00007f279308df17 in run (this=0x7f278f083480) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #6 platform_thread_wrap (arg=0x7f278f083480) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #7 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #8 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #9 0x0000000000000000 in ?? () Thread 8 (LWP 91386): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000000000015180 in ?? () #2 0x112e0be826d694b3 in ?? () #3 0x00007f278ef7f8f8 in ?? () #4 0x0000000000419560 in std::invalid_argument::~invalid_argument () #5 0x00000000004ed596 in __gthread_cond_timedwait (__abs_timeout=0x7f278cc785b0, __mutex=, __cond=0x7f278cc785b0) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f278cc785b0) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=0x7f278cc785b0) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=..., __lock=..., this=0x7f278cc785b0) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 AuditImpl::consume_events (this=0x7f278ef7f500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/auditd/src/audit.cc:417 #10 0x00007f279308df17 in run (this=0x7f278fe1c440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #11 platform_thread_wrap (arg=0x7f278fe1c440) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #12 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #13 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #14 0x0000000000000000 in ?? () Thread 7 (LWP 91385): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278efce378 in ?? () #2 0x112e0be826d694b3 in ?? () #3 0x0000000000419560 in std::invalid_argument::~invalid_argument () #4 0x00007f278efce350 in ?? () #5 0x0000000000494ad0 in __gthread_cond_timedwait (__abs_timeout=0x7f278d479590, __mutex=, __cond=0x7f278efce378) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=, this=0x7f278efce378) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=, this=0x7f278efce378) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_for > (__rtime=, __lock=, this=0x7f278efce378) at /usr/local/include/c++/7.3.0/condition_variable:138 #9 ExternalAuthManagerThread::run (this=0x7f278efce280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/external_auth_manager_thread.cc:199 #10 0x00007f27930997bf in Couchbase::Thread::thread_entry (this=0x7f278efce280) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:45 #11 0x00007f279308df17 in run (this=0x7f278fe1cbf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #12 platform_thread_wrap (arg=0x7f278fe1cbf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #13 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #14 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #15 0x0000000000000000 in ?? () Thread 6 (LWP 91384): #0 0x00007f27905c4bcd in __vsyslog_chk () from /usr/lib64/libc-2.17.so #1 0x00007f278dc7c9c0 in ?? () #2 0x00007f278dc7c700 in ?? () #3 0x0000000000000000 in ?? () Thread 5 (LWP 91383): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x0000000000419560 in std::invalid_argument::~invalid_argument () #2 0x00007f278fffcb40 in ?? () #3 0x00007f278e47b5b0 in ?? () #4 0x00007f278ef84118 in ?? () #5 0x00007f279412600a in __gthread_cond_timedwait (__abs_timeout=0x7f278e47b5b0, __mutex=, __cond=) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:129 #9 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __rtime=..., __lock=..., this=) at /usr/local/include/c++/7.3.0/condition_variable:145 #10 operator() (__closure=0xffffffffffffff9a) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/periodic_worker.h:38 #11 __invoke_impl&, std::chrono::seconds):: > (__f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:60 #12 __invoke&, std::chrono::seconds):: > (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #13 _M_invoke<0> (this=0xffffffffffffff9a) at /usr/local/include/c++/7.3.0/thread:234 #14 operator() (this=0xffffffffffffff9a) at /usr/local/include/c++/7.3.0/thread:243 #15 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv (this=0xffffffffffffff92) at /usr/local/include/c++/7.3.0/thread:186 #16 0x00007f2791088dcf in std::execute_native_thread_routine (__p=0x7f278fffcb40) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #17 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #18 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #19 0x0000000000000000 in ?? () Thread 4 (LWP 91382): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f278ffff138 in ?? () #2 0x16932074499a8cd1 in ?? () #3 0x00007f2794346c40 in vtable for fmt::v5::basic_memory_buffer > () from /opt/couchbase/lib/libmemcached_logger.so.1.0.0 #4 0x00007f278ec7c260 in ?? () #5 0x00007f279412694d in __gthread_cond_timedwait (__abs_timeout=0x7f278ec7c360, __mutex=, __cond=0x7f278ffff138) at /usr/local/include/c++/7.3.0/x86_64-pc-linux-gnu/bits/gthr-default.h:871 #6 __wait_until_impl > > (__atime=..., __lock=..., this=0x7f278ffff100) at /usr/local/include/c++/7.3.0/condition_variable:166 #7 wait_until > > (__atime=..., __lock=..., this=0x7f278ffff100) at /usr/local/include/c++/7.3.0/condition_variable:106 #8 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __atime=..., __lock=..., this=0x7f278ffff100) at /usr/local/include/c++/7.3.0/condition_variable:129 #9 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for(T&, std::chrono::milliseconds) [with T = spdlog::details::async_msg]:: > (__p=..., __rtime=, __lock=..., this=0x7f278ffff100) at /usr/local/include/c++/7.3.0/condition_variable:145 #10 dequeue_for (wait_duration=..., popped_item=..., this=0x7f278ffff110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/mpmc_blocking_q.h:61 #11 spdlog::details::thread_pool::process_next_msg_ (this=this@entry=0x7f278ffff110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:193 #12 0x00007f2794126ea0 in spdlog::details::thread_pool::worker_loop_ (this=0x7f278ffff110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:184 #13 0x00007f2791088dcf in std::execute_native_thread_routine (__p=0x7f278fe0a480) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #14 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #15 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #16 0x0000000000000000 in ?? () Thread 3 (LWP 91380): #0 0x00007f27908a5a82 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread-2.17.so #1 0x00007f279000a670 in ?? () #2 0x00007f278f9fde20 in ?? () #3 0x00007f278f9fd548 in ?? () #4 0x00007f279000a5c0 in ?? () #5 0x00007f27923a95a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #6 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #7 background_thread0_work (tsd=0xffffffffffffff92) at src/background_thread.c:452 #8 background_work (ind=, tsd=0xffffffffffffff92) at src/background_thread.c:490 #9 background_thread_entry (ind_arg=) at src/background_thread.c:522 #10 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #11 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #12 0x0000000000000000 in ?? () Thread 2 (LWP 91377): #0 0x00007f27905c9999 in fsetxattr () from /usr/lib64/libc-2.17.so #1 0x00007f279216c31a in epoll_wait (epfd=, events=, maxevents=, timeout=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll_sub.c:64 #2 0x00007f279216c7b5 in epoll_dispatch (base=0x281a6f0, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:462 #3 0x00007f27921644d2 in event_base_loop (base=0x281a6f0, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1947 #4 0x0000000000429f01 in memcached_main (argc=, argv=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:2522 #5 0x00007f27904fab15 in _nl_load_locale () from /usr/lib64/libc-2.17.so Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 1 (LWP 91686): #0 0x00007f2790636f45 in __memcmp_sse4_1 () from /usr/lib64/libc-2.17.so #1 0x00007f278bbde846 in compare (__n=, __s2=, __s1=) at /usr/local/include/c++/7.3.0/bits/char_traits.h:310 #2 compare (__str=..., this=0x7f2755e24580) at /usr/local/include/c++/7.3.0/bits/basic_string.h:2830 #3 compare (rhs=..., this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/storeddockey.h:121 #4 OrderItemsForDeDuplication::operator() (this=this@entry=0x7f275cff6ea0, i1=..., i2=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc:584 #5 0x00007f278bbeb5af in operator() >, __gnu_cxx::__normal_iterator >*, std::vector > > > > (__it=..., __val=..., this=0x7f275cff6ea0) at /usr/local/include/c++/7.3.0/bits/predefined_ops.h:215 #6 std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator >*, std::vector >, std::allocator > > > >, __gnu_cxx::__ops::_Val_comp_iter > (__last=..., __last@entry=..., __comp=...) at /usr/local/include/c++/7.3.0/bits/stl_algo.h:1828 #7 0x00007f278bbeb749 in std::__insertion_sort<__gnu_cxx::__normal_iterator >*, std::vector >, std::allocator > > > >, __gnu_cxx::__ops::_Iter_comp_iter > (__first=__first@entry=..., __last=..., __last@entry=..., __comp=...) at /usr/local/include/c++/7.3.0/bits/stl_algo.h:1855 #8 0x00007f278bbe7961 in __final_insertion_sort<__gnu_cxx::__normal_iterator >*, std::vector > > >, __gnu_cxx::__ops::_Iter_comp_iter > (__comp=..., __last=..., __first=...) at /usr/local/include/c++/7.3.0/bits/stl_algo.h:1890 #9 __sort<__gnu_cxx::__normal_iterator >*, std::vector > > >, __gnu_cxx::__ops::_Iter_comp_iter > (__comp=..., __last=..., __first=...) at /usr/local/include/c++/7.3.0/bits/stl_algo.h:1971 #10 sort<__gnu_cxx::__normal_iterator >*, std::vector > > >, OrderItemsForDeDuplication> (__last=..., __first=..., __comp=...) at /usr/local/include/c++/7.3.0/bits/stl_algo.h:4868 #11 KVStore::optimizeWrites (this=this@entry=0x7f2785734e00, items=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore.cc:591 #12 0x00007f278bb7b5bd in EPBucket::flushVBucket (this=0x7f278ef7c000, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:385 #13 0x00007f278bbcef7c in Flusher::flushVB (this=this@entry=0x7f27858e61c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:306 #14 0x00007f278bbcfeb9 in Flusher::step (this=0x7f27858e61c0, task=0x7f278f174b30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:207 #15 0x00007f278bbc9854 in ExecutorThread::run (this=0x7f278f1807a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:190 #16 0x00007f279308df17 in run (this=0x7f278f06dab0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58 #17 platform_thread_wrap (arg=0x7f278f06dab0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71 #18 0x00007f27908a1dc5 in start_thread () from /usr/lib64/libpthread-2.17.so #19 0x00007f27905cf21d in getspnam () from /usr/lib64/libc-2.17.so #20 0x0000000000000000 in ?? () -------------------------------------------------------------------------------- Module information: