Details
Critical Description
1. Load 550k documents on bucket default.
2. Create a 3 indexes w/ views and 1 gsi
3. Rebalance out 1 node.
3. Leave the cluster running overnight – Seeing a few memcached cores on one of the nodes.
Stack trace
[root@centos-64-x64 data]# gdb /opt/couchbase/bin/memcached core.memcached.15237
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 15258]
[New Thread 15245]
[New Thread 15244]
[New Thread 15239]
[New Thread 15237]
[New Thread 15241]
[New Thread 15243]
[New Thread 15249]
[New Thread 15238]
[New Thread 15255]
[New Thread 15256]
[New Thread 15240]
[New Thread 15247]
[New Thread 15250]
[New Thread 15246]
[New Thread 15254]
[New Thread 15248]
[New Thread 15252]
[New Thread 15257]
[New Thread 15253]
[New Thread 15251]
[New Thread 15242]
Reading symbols from /opt/couchbase/bin/../lib/libauditd.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libauditd.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...done.
Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /opt/couchbase/bin/../lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libdirutils.so.0.1.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libv8.so...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 8, Arithmetic exception.
#0 0x00007f524e3075ca in shouldContinueVisiting (this=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter_visitor.cc:190
190 /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter_visitor.cc: No such file or directory.
in /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter_visitor.cc
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.5.0-1335.x86_64
(gdb) t a a bt
Thread 22 (Thread 0x7f52521fc700 (LWP 15242)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423500, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423500, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460480) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f5249bfd700 (LWP 15251)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f52557684c0, mutex=0x7f5255768488, ms=583)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=192 '\300')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f5254253540) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253540) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041330) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f52487fb700 (LWP 15253)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f5255768340, mutex=0x7f5255768308, ms=2000)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
--Type <return> to continue, or q <return> to quit--
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=144 '\220')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f5254253700) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253700) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041360) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f5245ff7700 (LWP 15257)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f5255768640, mutex=0x7f5255768608, ms=2000)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768600, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768600, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768600, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768600, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=219 '\333')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f5254253a80) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253a80) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b0413a0) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f52491fc700 (LWP 15252)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f52557684c0, mutex=0x7f5255768488, ms=1184)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
--Type <return> to continue, or q <return> to quit--
#2 0x00007f524e371d45 in wait (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=192 '\300')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f5254253620) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253620) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041350) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f524c048700 (LWP 15248)):
#0 0x00007f525743eb8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f5257473d64 in usleep () from /lib64/libc.so.6
#2 0x00007f524e34c895 in updateStatsThread (arg=0x7f5255429140) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/memory_tracker.cc:36
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f52554606a0) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f5247dfa700 (LWP 15254)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f5255768340, mutex=0x7f5255768308, ms=2000)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=144 '\220')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
--Type <return> to continue, or q <return> to quit--
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f52542537e0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f52542537e0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041370) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f524f9f8700 (LWP 15246)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423f00, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423f00, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460440) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f524a5fe700 (LWP 15250)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f52557684c0, mutex=0x7f5255768488, ms=583)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=192 '\300')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f5254253460) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253460) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041340) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f524eff7700 (LWP 15247)):
--Type <return> to continue, or q <return> to quit--
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255424180, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255424180, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460260) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f52535fe700 (LWP 15240)):
#0 0x00007f52582dc43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f5259dc5e0d in consume_events (arg=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/memcached/auditd/src/auditd.cc:59
#2 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460040) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#3 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f52469f8700 (LWP 15256)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f5255768340, mutex=0x7f5255768308, ms=2000)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=144 '\220')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f52542539a0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f52542539a0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041390) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f52473f9700 (LWP 15255)):
--Type <return> to continue, or q <return> to quit--
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f5255768340, mutex=0x7f5255768308, ms=2000)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768300, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768300, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=144 '\220')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f52542538c0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f52542538c0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041380) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f52553ff700 (LWP 15238)):
#0 0x00007f525746d60d in read () from /lib64/libc.so.6
#1 0x00007f5257403f68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f5257405a6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f52573fa14a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f52573f8fa9 in fgets () from /lib64/libc.so.6
#5 0x00007f5255dda961 in check_stdin_thread (arg=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460060) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#7 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f524afff700 (LWP 15249)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f52557684c0, mutex=0x7f5255768488, ms=584)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f524e371d45 in wait (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/syncobject.h:74
#3 TaskQueue::_doSleep (this=0x7f5255768480, t=...) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:94
#4 0x00007f524e373ceb in _fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
)
--Type <return> to continue, or q <return> to quit--
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:117
#5 TaskQueue::fetchNextTask (this=0x7f5255768480, thread=..., toSleep=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/taskqueue.cc:161
#6 0x00007f524e33abf6 in ExecutorPool::_nextTask (this=0x7f524b05b800, t=..., tick=192 '\300')
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:214
#7 0x00007f524e33acd1 in ExecutorPool::nextTask (this=0x7f524b05b800, t=..., tick=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorpool.cc:229
#8 0x00007f524e34e148 in ExecutorThread::run (this=0x7f5254253380) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:78
#9 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253380) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#10 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b041320) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#11 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f52517fb700 (LWP 15243)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423780, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423780, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460470) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f5252bfd700 (LWP 15241)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423280, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423280, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f52554604a0) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f525a1e2820 (LWP 15237)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423000, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
--Type <return> to continue, or q <return> to quit--
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423000, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x000000000040b42f in main (argc=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/memcached/daemon/memcached.c:8526
Thread 4 (Thread 0x7f5253fff700 (LWP 15239)):
#0 0x00007f52582dc7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f52597b7b85 in cb_cond_timedwait (cond=0x7f52549fe940, mutex=0x7f52549fe980, ms=19000)
at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:156
#2 0x00007f52547fc793 in logger_thead_main (arg=0x7f5255429500)
at /home/couchbase/jenkins/workspace/sherlock-unix/memcached/extensions/loggers/file_logger.c:498
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460050) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f5250dfa700 (LWP 15244)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423a00, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423a00, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460450) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f52503f9700 (LWP 15245)):
#0 0x00007f525747af03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f5258d59e93 in epoll_dispatch (base=0x7f5255423c80, tv=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:407
#2 0x00007f5258d46f42 in event_base_loop (base=0x7f5255423c80, flags=0)
at /home/couchbase/jenkins/workspace/cbdeps-build/label/centos6/release/sherlock/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1625
#3 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f5255460460) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#4 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f525747a90d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f52455f6700 (LWP 15258)):
#0 0x00007f524e3075ca in shouldContinueVisiting (this=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter_visitor.cc:190
--Type <return> to continue, or q <return> to quit--
#1 DefragmentVisitor::visit (this=Unhandled dwarf expression opcode 0xf3
) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter_visitor.cc:109
#2 0x00007f524e356585 in HashTable::pauseResumeVisit (this=0x7f524049ec08, visitor=..., start_pos=...)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/stored-value.cc:518
#3 0x00007f524e307466 in DefragmentVisitor::visit (this=0x7f51b383dec0, vbucket_id=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter_visitor.cc:77
#4 0x00007f524e309315 in EventuallyPersistentStore::pauseResumeVisit (this=0x7f525571c000, visitor=..., start_pos=Unhandled dwarf expression opcode 0xf3
)
at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/ep.cc:3454
#5 0x00007f524e306511 in DefragmenterTask::run (this=0x7f524b0963c0) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/defragmenter.cc:69
#6 0x00007f524e34e32f in ExecutorThread::run (this=0x7f5254253b60) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:110
#7 0x00007f524e34e9a9 in launch_executor_thread (arg=0x7f5254253b60) at /home/couchbase/jenkins/workspace/sherlock-unix/ep-engine/src/executorthread.cc:34
#8 0x00007f52597b78aa in platform_thread_wrap (arg=0x7f524b0413b0) at /home/couchbase/jenkins/workspace/sherlock-unix/platform/src/cb_pthreads.c:19
#9 0x00007f52582d8851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f525747a90d in clone () from /lib64/libc.so.6