/etc/init.d/couchbase-server stop not able to gracefully stop couchbase server, memcached exits w/ abort - "134"

On a toy-build spun from master branch,

issue a "/etc/init.d/couchbase-server stop"
Couchbase node does not exit gracefully.

Port memcached exits with 134 - Sig Abort.

Attaching core.memcached.

[root@soursop-s11202 couchbase]# gdb /opt/couchbase/bin/memcached core.1368
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 1401]
[New Thread 1368]
[New Thread 1406]
[New Thread 1403]
[New Thread 1384]
[New Thread 1387]
[New Thread 1393]
[New Thread 1395]
[New Thread 1386]
[New Thread 1394]
[New Thread 1378]
[New Thread 1389]
[New Thread 1388]
[New Thread 1399]
[New Thread 1402]
[New Thread 1398]
[New Thread 1400]
[New Thread 1397]
[New Thread 1392]
[New Thread 1391]
[New Thread 1385]
[New Thread 1377]
[New Thread 1404]
[New Thread 1396]
[New Thread 1390]
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007f33e8ced8a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-0.0.0-704.x86_64
(gdb) t a a bt

Thread 25 (Thread 0x7f33dd48c700 (LWP 1390)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd7c0) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d0598,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d0598, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d0598,
now=...) at src/scheduler.cc:153
#7 0x00007f33e3669b1c in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:301
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d0510) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f33d9886700 (LWP 1396)):
#0 0x00007f33e8db309f in __lll_unlock_wake_private () from /lib64/libc.so.6
#1 0x00007f33e8d22625 in _L_unlock_112 () from /lib64/libc.so.6
#2 0x00007f33e8d22593 in fputs () from /lib64/libc.so.6
#3 0x00007f33e7717a13 in logger_log (severity=EXTENSION_LOG_WARNING,
client_cookie=<value optimized out>, fmt=0x7f33e36c8fff "(%s) %s")
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:266
#4 0x00007f33e3644aae in LOG (severity=EXTENSION_LOG_WARNING,
fmt=0x7f33e36c7198 "Total memory use reached to the low water mark, stop warmup")
at src/ep_engine.cc:1177
#5 0x00007f33e362eae0 in EventuallyPersistentStore::maybeEnableTraffic (this=0x513e000)
--Type <return> to continue, or q <return> to quit--
at src/ep.cc:2459
#6 0x00007f33e3694ff7 in LoadStorageKVPairCallback::callback (this=0x1443b940, val=...)
at src/warmup.cc:285
#7 0x00007f33e36a1efb in CouchKVStore::recordDbDump (db=0xef8fb6b0, docinfo=0x52250a0,
ctx=0x7f33d9885b00) at src/couch-kvstore/couch-kvstore.cc:1308
#8 0x00007f33e33e859d in lookup_callback (rq=<value optimized out>, k=0x7f33d98852c0,
v=0x7f33d98852b0) at src/couch_db.cc:696
#9 0x00007f33e33e7294 in btree_lookup_inner (rq=0x7f33d9885570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:78
#10 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d9885570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#11 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d9885570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#12 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d9885570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#13 0x00007f33e33e97f6 in couchstore_changes_since (db=<value optimized out>,
since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>,
ctx=<value optimized out>) at src/couch_db.cc:739
#14 0x00007f33e36a5d51 in CouchKVStore::loadDB (this=0x5187500,
cb=std::tr1::shared_ptr (count 5) 0x1443b940, keysOnly=false, vbids=<value optimized out>,
options=4) at src/couch-kvstore/couch-kvstore.cc:952
#15 0x00007f33e36a696d in CouchKVStore::dump (this=<value optimized out>,
vbids=<value optimized out>, cb=<value optimized out>)
at src/couch-kvstore/couch-kvstore.cc:760
#16 0x00007f33e3695fef in Warmup::loadDataforShard (this=0x510a1e0, shardId=2)
at src/warmup.cc:666
#17 0x00007f33e3697e27 in WarmupLoadingData::run() () from /opt/couchbase/lib/memcached/ep.so
#18 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d0e10) at src/scheduler.cc:97
#19 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#20 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f33e8da390d in clone () from /lib64/libc.so.6
--Type <return> to continue, or q <return> to quit--

Thread 23 (Thread 0x7f33d487e700 (LWP 1404)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0x3c93c831,
nbyte=8192, offset=74866688) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0x3c86d060, buf=0x3c93c831, sz=8192, off=74866688)
at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007f33e33ed489 in load_buffer_from (handle=0x13f3ecf0, buf=<value optimized out>,
nbyte=8, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0x13f3ecf0, buf=<value optimized out>, nbyte=8,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e99dc in read_skipping_prefixes (file=0x50dddc0, pos=74866989,
ret_ptr=0x7f33d487d238, max_header_size=0) at src/couch_file_read.cc:79
#6 pread_bin_internal (file=0x50dddc0, pos=74866989, ret_ptr=0x7f33d487d238, max_header_size=0)
at src/couch_file_read.cc:111
#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33d487d2e8) at src/couch_file_read.cc:157
#8 0x00007f33e33e6f9c in btree_lookup_inner (rq=0x7f33d487d570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:22
#9 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d487d570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#10 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d487d570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#11 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d487d570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#12 0x00007f33e33e97f6 in couchstore_changes_since (db=<value optimized out>,
since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>,
ctx=<value optimized out>) at src/couch_db.cc:739
#13 0x00007f33e36a5d51 in CouchKVStore::loadDB (this=0x5187500,
cb=std::tr1::shared_ptr (count 4) 0x52d5380, keysOnly=false, vbids=<value optimized out>,
options=4) at src/couch-kvstore/couch-kvstore.cc:952
#14 0x00007f33e36a696d in CouchKVStore::dump (this=<value optimized out>,
--Type <return> to continue, or q <return> to quit--
vbids=<value optimized out>, cb=<value optimized out>)
at src/couch-kvstore/couch-kvstore.cc:760
#15 0x00007f33e3695fef in Warmup::loadDataforShard (this=0x510a1e0, shardId=3)
at src/warmup.cc:666
#16 0x00007f33e3697e27 in WarmupLoadingData::run() () from /opt/couchbase/lib/memcached/ep.so
#17 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d1050) at src/scheduler.cc:97
#18 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#19 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f33e8319700 (LWP 1377)):
#0 0x00007f33e8d9660d in read () from /lib64/libc.so.6
#1 0x00007f33e8d2cf68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f33e8d2ea6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f33e8d2314a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f33e8d21fa9 in fgets () from /lib64/libc.so.6
#5 0x00007f33e831a8c1 in check_stdin_thread (arg=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007f33e9ada76f in platform_thread_wrap (arg=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/platform/src/cb_pthreads.c:18
#7 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f33e0691700 (LWP 1385)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0xca76831, nbyte=8192,
offset=98025472) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0x12f174a0, buf=0xca76831, sz=8192, off=98025472)
at src/couch-kvstore/couch-fs-stats.cc:88
--Type <return> to continue, or q <return> to quit--
#3 0x00007f33e33ed489 in load_buffer_from (handle=0x532eb40, buf=<value optimized out>,
nbyte=8, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0x532eb40, buf=<value optimized out>, nbyte=8,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e99dc in read_skipping_prefixes (file=0x50dd7a0, pos=98029770,
ret_ptr=0x7f33e0690238, max_header_size=0) at src/couch_file_read.cc:79
#6 pread_bin_internal (file=0x50dd7a0, pos=98029770, ret_ptr=0x7f33e0690238, max_header_size=0)
at src/couch_file_read.cc:111
#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33e06902e8) at src/couch_file_read.cc:157
#8 0x00007f33e33e6f9c in btree_lookup_inner (rq=0x7f33e0690570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:22
#9 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33e0690570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#10 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33e0690570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#11 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33e0690570, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#12 0x00007f33e33e97f6 in couchstore_changes_since (db=<value optimized out>,
since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>,
ctx=<value optimized out>) at src/couch_db.cc:739
#13 0x00007f33e36a5d51 in CouchKVStore::loadDB (this=0x5187500,
cb=std::tr1::shared_ptr (count 4) 0x53149c0, keysOnly=false, vbids=<value optimized out>,
options=4) at src/couch-kvstore/couch-kvstore.cc:952
#14 0x00007f33e36a696d in CouchKVStore::dump (this=<value optimized out>,
vbids=<value optimized out>, cb=<value optimized out>)
at src/couch-kvstore/couch-kvstore.cc:760
#15 0x00007f33e3695fef in Warmup::loadDataforShard (this=0x510a1e0, shardId=1)
at src/warmup.cc:666
#16 0x00007f33e3697e27 in WarmupLoadingData::run() () from /opt/couchbase/lib/memcached/ep.so
#17 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d0120) at src/scheduler.cc:97
#18 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
--Type <return> to continue, or q <return> to quit--
at src/scheduler.cc:38
#19 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f33dca8b700 (LWP 1391)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0xf2510431,
nbyte=8192, offset=211165184) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0xfc932d60, buf=0xf2510431, sz=8192, off=211165184)
at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007f33e33ed489 in load_buffer_from (handle=0xf253ec60, buf=<value optimized out>,
nbyte=8, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0xf253ec60, buf=<value optimized out>, nbyte=8,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e99dc in read_skipping_prefixes (file=0x521f1f0, pos=211172854,
ret_ptr=0x7f33dca8a258, max_header_size=0) at src/couch_file_read.cc:79
#6 pread_bin_internal (file=0x521f1f0, pos=211172854, ret_ptr=0x7f33dca8a258,
max_header_size=0) at src/couch_file_read.cc:111
#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33dca8a2f8) at src/couch_file_read.cc:157
#8 0x00007f33e33e60d8 in modify_node (rq=0x7f33dca8a5e0, nptr=0x113fdedb4, start=12, end=15,
dst=0x114183ac0) at src/btree_modify.cc:341
#9 0x00007f33e33e62ba in modify_node (rq=0x7f33dca8a5e0, nptr=0x114183a90, start=12, end=18,
dst=0x1142aa0c8) at src/btree_modify.cc:482
#10 0x00007f33e33e62ba in modify_node (rq=0x7f33dca8a5e0, nptr=0xfd33e540, start=11, end=296,
dst=0x1142aa010) at src/btree_modify.cc:482
#11 0x00007f33e33e6d07 in modify_btree (rq=0x7f33dca8a5e0, root=0xfd33e540,
errcode=0x7f33dca8a6ec) at src/btree_modify.cc:640
#12 0x00007f33e33eb01d in update_indexes (db=<value optimized out>, docs=<value optimized out>,
infos=0xe7f78000, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:243
#13 couchstore_save_documents (db=<value optimized out>, docs=<value optimized out>,
--Type <return> to continue, or q <return> to quit--
infos=0xe7f78000, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:382
#14 0x00007f33e36a9a69 in CouchKVStore::saveDocs (this=0x5155500, vbid=90, rev=21,
docs=0xe7f7aa00, docinfos=0xe7f78000, docCount=197)
at src/couch-kvstore/couch-kvstore.cc:1415
#15 0x00007f33e36aa31b in CouchKVStore::commit2couchstore (this=0x5155500)
at src/couch-kvstore/couch-kvstore.cc:1352
#16 0x00007f33e36aa507 in CouchKVStore::commit (this=0x5155500)
at src/couch-kvstore/couch-kvstore.cc:792
#17 0x00007f33e3638c5e in EventuallyPersistentStore::flushVBucket (this=0x513e000, vbid=90)
at src/ep.cc:2221

#18 0x00007f33e365e60b in doFlush (this=0x8bd2c0, tid=12) at src/flusher.cc:230
#19 Flusher::step (this=0x8bd2c0, tid=12) at src/flusher.cc:161
#20 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d0480) at src/scheduler.cc:97
#21 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#22 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f33dc08a700 (LWP 1392)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0x6c34c31, nbyte=8192,
offset=213319680) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0xf255ad60, buf=0x6c34c31, sz=8192, off=213319680)
at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007f33e33ed489 in load_buffer_from (handle=0x113a851d0, buf=<value optimized out>,
nbyte=8, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0x113a851d0, buf=<value optimized out>, nbyte=8,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e99dc in read_skipping_prefixes (file=0xe7f6bab0, pos=213325400,
ret_ptr=0x7f33dc088f98, max_header_size=0) at src/couch_file_read.cc:79
#6 pread_bin_internal (file=0xe7f6bab0, pos=213325400, ret_ptr=0x7f33dc088f98,
--Type <return> to continue, or q <return> to quit--
max_header_size=0) at src/couch_file_read.cc:111
#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33dc089038) at src/couch_file_read.cc:157
#8 0x00007f33e33e60d8 in modify_node (rq=0x7f33dc089590, nptr=0x1142816bc, start=144, end=146,
dst=0x114281384) at src/btree_modify.cc:341

#9 0x00007f33e33e62ba in modify_node (rq=0x7f33dc089590, nptr=0x114281354, start=144, end=146,
dst=0x11428129c) at src/btree_modify.cc:482
#10 0x00007f33e33e6bc5 in modify_node (rq=0x7f33dc089590, nptr=0x11428126c, start=144, end=146,
dst=0x1142811b4) at src/btree_modify.cc:446
#11 0x00007f33e33e6bc5 in modify_node (rq=0x7f33dc089590, nptr=0x114281184, start=144, end=146,
dst=0x11428104c) at src/btree_modify.cc:446
#12 0x00007f33e33e6bc5 in modify_node (rq=0x7f33dc089590, nptr=0x11428101c, start=144, end=146,
dst=0x114280f64) at src/btree_modify.cc:446
#13 0x00007f33e33e6bc5 in modify_node (rq=0x7f33dc089590, nptr=0x114280f34, start=144, end=146,
dst=0xe7f6c0c8) at src/btree_modify.cc:446
#14 0x00007f33e33e62ba in modify_node (rq=0x7f33dc089590, nptr=0x51c7880, start=144, end=350,
dst=0xe7f6c010) at src/btree_modify.cc:482
#15 0x00007f33e33e6d07 in modify_btree (rq=0x7f33dc089590, root=0x51c7880,
errcode=0x7f33dc0896ec) at src/btree_modify.cc:640
#16 0x00007f33e33eaed1 in update_indexes (db=<value optimized out>, docs=<value optimized out>,
infos=0x113f70b00, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:218
#17 couchstore_save_documents (db=<value optimized out>, docs=<value optimized out>,
infos=0x113f70b00, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:382
#18 0x00007f33e36a9a69 in CouchKVStore::saveDocs (this=0x5154e00, vbid=81, rev=22,
docs=0x113924100, docinfos=0x113f70b00, docCount=175)
at src/couch-kvstore/couch-kvstore.cc:1415
#19 0x00007f33e36aa31b in CouchKVStore::commit2couchstore (this=0x5154e00)
at src/couch-kvstore/couch-kvstore.cc:1352
#20 0x00007f33e36aa507 in CouchKVStore::commit (this=0x5154e00)
at src/couch-kvstore/couch-kvstore.cc:792
--Type <return> to continue, or q <return> to quit--
#21 0x00007f33e3638c5e in EventuallyPersistentStore::flushVBucket (this=0x513e000, vbid=81)
at src/ep.cc:2221
#22 0x00007f33e365e60b in doFlush (this=0x8bcdc0, tid=10) at src/flusher.cc:230
#23 Flusher::step (this=0x8bcdc0, tid=10) at src/flusher.cc:161
#24 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d03f0) at src/scheduler.cc:97
#25 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#26 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#27 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f33d8e85700 (LWP 1397)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0x6fdc0431,
nbyte=8192, offset=99819520) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0x113c81e40, buf=0x6fdc0431, sz=8192, off=99819520)
at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007f33e33ed489 in load_buffer_from (handle=0x113bba420, buf=<value optimized out>,
nbyte=538, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0x113bba420, buf=<value optimized out>, nbyte=538,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e9b0e in read_skipping_prefixes (file=0x521eb60, pos=99819521,
ret_ptr=0x7f33d8e841a8, max_header_size=<value optimized out>) at src/couch_file_read.cc:79

#6 pread_bin_internal (file=0x521eb60, pos=99819521, ret_ptr=0x7f33d8e841a8,
max_header_size=<value optimized out>) at src/couch_file_read.cc:128
#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33d8e84248) at src/couch_file_read.cc:157
#8 0x00007f33e33e60d8 in modify_node (rq=0x7f33d8e845e0, nptr=0x113b87654, start=4, end=5,
dst=0x113b86ba4) at src/btree_modify.cc:341
#9 0x00007f33e33e62ba in modify_node (rq=0x7f33d8e845e0, nptr=0x113b86b74, start=4, end=5,
dst=0x113b864d8) at src/btree_modify.cc:482
#10 0x00007f33e33e62ba in modify_node (rq=0x7f33d8e845e0, nptr=0x113b864a8, start=4, end=6,
dst=0x1142200c8) at src/btree_modify.cc:482
--Type <return> to continue, or q <return> to quit--
#11 0x00007f33e33e62ba in modify_node (rq=0x7f33d8e845e0, nptr=0x113da1b40, start=4, end=293,
dst=0x114220010) at src/btree_modify.cc:482
#12 0x00007f33e33e6d07 in modify_btree (rq=0x7f33d8e845e0, root=0x113da1b40,
errcode=0x7f33d8e846ec) at src/btree_modify.cc:640
#13 0x00007f33e33eb01d in update_indexes (db=<value optimized out>, docs=<value optimized out>,
infos=0x113a6b500, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:243
#14 couchstore_save_documents (db=<value optimized out>, docs=<value optimized out>,
infos=0x113a6b500, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:382
#15 0x00007f33e36a9a69 in CouchKVStore::saveDocs (this=0x5186380, vbid=75, rev=22,
docs=0x113a6bc00, docinfos=0x113a6b500, docCount=194)
at src/couch-kvstore/couch-kvstore.cc:1415
#16 0x00007f33e36aa31b in CouchKVStore::commit2couchstore (this=0x5186380)
at src/couch-kvstore/couch-kvstore.cc:1352
#17 0x00007f33e36aa507 in CouchKVStore::commit (this=0x5186380)
at src/couch-kvstore/couch-kvstore.cc:792
#18 0x00007f33e3638c5e in EventuallyPersistentStore::flushVBucket (this=0x513e000, vbid=75)
at src/ep.cc:2221
#19 0x00007f33e365e60b in doFlush (this=0x8bd180, tid=13) at src/flusher.cc:230
#20 Flusher::step (this=0x8bd180, tid=13) at src/flusher.cc:161
#21 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d0d80) at src/scheduler.cc:97
#22 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#23 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#24 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f33d7082700 (LWP 1400)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0xf27fac31,
nbyte=8192, offset=182525952) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0xc1cad2e0, buf=0xf27fac31, sz=8192, off=182525952)
--Type <return> to continue, or q <return> to quit--
at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007f33e33ed489 in load_buffer_from (handle=0x111b69830, buf=<value optimized out>,
nbyte=8, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0x111b69830, buf=<value optimized out>, nbyte=8,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e99dc in read_skipping_prefixes (file=0x5225650, pos=182530412,
ret_ptr=0x7f33d7080f98, max_header_size=0) at src/couch_file_read.cc:79
#6 pread_bin_internal (file=0x5225650, pos=182530412, ret_ptr=0x7f33d7080f98,
max_header_size=0) at src/couch_file_read.cc:111

#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33d7081038) at src/couch_file_read.cc:157
#8 0x00007f33e33e60d8 in modify_node (rq=0x7f33d7081590, nptr=0x113e40ad8, start=134, end=136,
dst=0x113e40a20) at src/btree_modify.cc:341
#9 0x00007f33e33e62ba in modify_node (rq=0x7f33d7081590, nptr=0x113e409f0, start=134, end=136,
dst=0x113e40938) at src/btree_modify.cc:482
#10 0x00007f33e33e62ba in modify_node (rq=0x7f33d7081590, nptr=0x113e40908, start=134, end=136,
dst=0x114282c98) at src/btree_modify.cc:482
#11 0x00007f33e33e62ba in modify_node (rq=0x7f33d7081590, nptr=0x114282c68, start=134, end=138,
dst=0x114282bb0) at src/btree_modify.cc:482
#12 0x00007f33e33e62ba in modify_node (rq=0x7f33d7081590, nptr=0x114282b80, start=122, end=140,
dst=0x1139561b0) at src/btree_modify.cc:482
#13 0x00007f33e33e62ba in modify_node (rq=0x7f33d7081590, nptr=0x113956180, start=122, end=214,
dst=0x1139560c8) at src/btree_modify.cc:482
#14 0x00007f33e33e62ba in modify_node (rq=0x7f33d7081590, nptr=0x1138d6c40, start=0, end=396,
dst=0x113956010) at src/btree_modify.cc:482
#15 0x00007f33e33e6d07 in modify_btree (rq=0x7f33d7081590, root=0x1138d6c40,
errcode=0x7f33d70816ec) at src/btree_modify.cc:640
#16 0x00007f33e33eaed1 in update_indexes (db=<value optimized out>, docs=<value optimized out>,
infos=0x113c04a00, numdocs=<value optimized out>, options=<value optimized out>)
at src/couch_save.cc:218
#17 couchstore_save_documents (db=<value optimized out>, docs=<value optimized out>,
infos=0x113c04a00, numdocs=<value optimized out>, options=<value optimized out>)
--Type <return> to continue, or q <return> to quit--
at src/couch_save.cc:382
#18 0x00007f33e36a9a69 in CouchKVStore::saveDocs (this=0x5154000, vbid=80, rev=21,
docs=0x113c03c00, docinfos=0x113c04a00, docCount=198)
at src/couch-kvstore/couch-kvstore.cc:1415
#19 0x00007f33e36aa31b in CouchKVStore::commit2couchstore (this=0x5154000)
at src/couch-kvstore/couch-kvstore.cc:1352
#20 0x00007f33e36aa507 in CouchKVStore::commit (this=0x5154000)
at src/couch-kvstore/couch-kvstore.cc:792
#21 0x00007f33e3638c5e in EventuallyPersistentStore::flushVBucket (this=0x513e000, vbid=80)
at src/ep.cc:2221
#22 0x00007f33e365e60b in doFlush (this=0x8bcf00, tid=11) at src/flusher.cc:230
#23 Flusher::step (this=0x8bcf00, tid=11) at src/flusher.cc:161
#24 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d0ea0) at src/scheduler.cc:97
#25 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#26 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#27 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f33d8484700 (LWP 1398)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd7c0) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d1048,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d1048, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d1048,
now=...) at src/scheduler.cc:153
#7 0x00007f33e3669b1c in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:301
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d0fc0) at src/scheduler.cc:82
--Type <return> to continue, or q <return> to quit--
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f33d5c80700 (LWP 1402)):

#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd7c0) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d11f8,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d11f8, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d11f8,
now=...) at src/scheduler.cc:153
#7 0x00007f33e36699b1 in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:281
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d1170) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f33d7a83700 (LWP 1399)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd040) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd040, task=..., waketime=..., taskType=@0x50d0fb8,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd040, task=..., waketime=..., taskType=@0x50d0fb8, now=...)
--Type <return> to continue, or q <return> to quit--
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd040, task=..., waketime=..., taskType=@0x50d0fb8,
now=...) at src/scheduler.cc:153
#7 0x00007f33e3669b1c in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:301
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d0f30) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f33de88e700 (LWP 1388)):
#0 0x00007f33e94681e5 in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x00007f33e94646f7 in _L_unlock_657 () from /lib64/libpthread.so.0
#2 0x00007f33e946465f in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b96 in Mutex::release (this=0x5181bb0) at src/mutex.cc:93
#4 0x00007f33e36697d6 in unlock (this=0x8bd680, task=..., waketime=<value optimized out>,
taskType=@0x50d02c8, now=...) at src/locks.h:76
#5 ~LockHolder (this=0x8bd680, task=..., waketime=<value optimized out>, taskType=@0x50d02c8,
now=...) at src/locks.h:59
#6 doneWork (this=0x8bd680, task=..., waketime=<value optimized out>, taskType=@0x50d02c8,
now=...) at src/scheduler.cc:381
#7 TaskQueue::fetchNextTask (this=0x8bd680, task=..., waketime=<value optimized out>,
taskType=@0x50d02c8, now=...) at src/scheduler.cc:164
#8 0x00007f33e3669b1c in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:301
#9 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d0240) at src/scheduler.cc:82
#10 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#11 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f33e8da390d in clone () from /lib64/libc.so.6

--Type <return> to continue, or q <return> to quit--
Thread 12 (Thread 0x7f33dde8d700 (LWP 1389)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0

#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd7c0) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d0628,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d0628, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d0628,
now=...) at src/scheduler.cc:153
#7 0x00007f33e3669b1c in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:301
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d05a0) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f33e7715700 (LWP 1378)):
#0 0x00007f33e94657bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f33e9ada533 in cb_cond_timedwait (cond=0x7f33e7918800, mutex=0x7f33e79187c0,
ms=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/platform/src/cb_pthreads.c:133
#2 0x00007f33e7717d4d in logger_thead_main (arg=0x8b2040)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:363
#3 0x00007f33e9ada76f in platform_thread_wrap (arg=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/platform/src/cb_pthreads.c:18
#4 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#5 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f33dac88700 (LWP 1394)):
#0 0x00007f33e94657bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f33e36465d2 in wait (this=0x511a000) at src/syncobject.h:77
#2 wait (this=0x511a000) at src/syncobject.h:93
#3 wait (this=0x511a000) at src/tapconnmap.h:225
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x511a000)
at src/ep_engine.cc:3389
#5 0x00007f33e36466b3 in EvpNotifyPendingConns (arg=0x511a000) at src/ep_engine.cc:1139
#6 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f33dfc90700 (LWP 1386)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd680) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd680, task=..., waketime=..., taskType=@0x50d03e8,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd680, task=..., waketime=..., taskType=@0x50d03e8, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd680, task=..., waketime=..., taskType=@0x50d03e8,
now=...) at src/scheduler.cc:153
#7 0x00007f33e36699b1 in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:281
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d0360) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

--Type <return> to continue, or q <return> to quit--
Thread 8 (Thread 0x7f33da287700 (LWP 1395)):
#0 0x00007f33e94657bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1 0x00007f33e362a548 in wait (this=0x50d07e0, d=...) at src/syncobject.h:77
#2 IdleTask::run (this=0x50d07e0, d=...) at src/dispatcher.cc:344
#3 0x00007f33e362d0aa in Dispatcher::run (this=0x50e6a80) at src/dispatcher.cc:186

#4 0x00007f33e362d86d in launch_dispatcher_thread (arg=0x50e6a80) at src/dispatcher.cc:30
#5 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f33db689700 (LWP 1393)):
#0 0x00007f33e9468e53 in pread64 () from /lib64/libpthread.so.0
#1 0x00007f33e33f6458 in couch_pread (handle=<value optimized out>, buf=0x55822431,
nbyte=8192, offset=35807232) at src/os.c:38
#2 0x00007f33e36afd4d in cfs_pread (h=0x144472a0, buf=0x55822431, sz=8192, off=35807232)
at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007f33e33ed489 in load_buffer_from (handle=0x522a630, buf=<value optimized out>,
nbyte=543, offset=<value optimized out>) at src/iobuffer.cc:157
#4 buffered_pread (handle=0x522a630, buf=<value optimized out>, nbyte=543,
offset=<value optimized out>) at src/iobuffer.cc:293
#5 0x00007f33e33e9b0e in read_skipping_prefixes (file=0x50dce70, pos=35807233,
ret_ptr=0x7f33db6881d8, max_header_size=<value optimized out>) at src/couch_file_read.cc:79
#6 pread_bin_internal (file=0x50dce70, pos=35807233, ret_ptr=0x7f33db6881d8,
max_header_size=<value optimized out>) at src/couch_file_read.cc:128
#7 0x00007f33e33e9bd7 in pread_compressed (file=<value optimized out>,
pos=<value optimized out>, ret_ptr=0x7f33db688288) at src/couch_file_read.cc:157
#8 0x00007f33e33e6f9c in btree_lookup_inner (rq=0x7f33db688510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:22
#9 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33db688510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#10 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33db688510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#11 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33db688510, diskpos=<value optimized out>,
--Type <return> to continue, or q <return> to quit--
current=0, end=1) at src/btree_read.cc:52
#12 0x00007f33e33e97f6 in couchstore_changes_since (db=<value optimized out>,
since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>,
ctx=<value optimized out>) at src/couch_db.cc:739

#13 0x00007f33e36a5d51 in CouchKVStore::loadDB (this=0x5187500, cb=
std::tr1::shared_ptr (count 4) 0x522a690, keysOnly=false, vbids=<value optimized out>,
options=0) at src/couch-kvstore/couch-kvstore.cc:952
#14 0x00007f33e36a6883 in CouchKVStore::dump (this=0x5187500, vb=201, cb=<value optimized out>)
at src/couch-kvstore/couch-kvstore.cc:767
#15 0x00007f33e361c675 in BackfillDiskLoad::run (this=0xf2e5fb00) at src/backfill.cc:79
#16 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d0870) at src/scheduler.cc:97
#17 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#18 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f33df28f700 (LWP 1387)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd680) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd680, task=..., waketime=..., taskType=@0x50d0358,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd680, task=..., waketime=..., taskType=@0x50d0358, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd680, task=..., waketime=..., taskType=@0x50d0358,
now=...) at src/scheduler.cc:153
#7 0x00007f33e3669b1c in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:301
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d02d0) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
--Type <return> to continue, or q <return> to quit--
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f33e1092700 (LWP 1384)):
#0 0x00007f33e8d67b8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f33e8d9cd64 in usleep () from /lib64/libc.so.6

#2 0x00007f33e3663c25 in updateStatsThread (arg=0x8b2400) at src/memory_tracker.cc:36
#3 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f33d527f700 (LWP 1403)):
#0 0x00007f33e9468054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f33e9463388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f33e9463257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f33e3664b1a in Mutex::acquire (this=0x8bd7c0) at src/mutex.cc:80
#4 0x00007f33e36695f1 in lock (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d1168,
now=...) at src/locks.h:66
#5 LockHolder (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d1168, now=...)
at src/locks.h:44
#6 TaskQueue::fetchNextTask (this=0x8bd7c0, task=..., waketime=..., taskType=@0x50d1168,
now=...) at src/scheduler.cc:153
#7 0x00007f33e36699b1 in ExecutorPool::nextTask (this=0x5181ba0, t=...,
tick=<value optimized out>) at src/scheduler.cc:281
#8 0x00007f33e3669c68 in ExecutorThread::run (this=0x50d10e0) at src/scheduler.cc:82
#9 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#10 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f33d347c700 (LWP 1406)):
#0 0x00007f33e94657bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f33e362a548 in wait (this=0x50d14d0, d=...) at src/syncobject.h:77
--Type <return> to continue, or q <return> to quit--
#2 IdleTask::run (this=0x50d14d0, d=...) at src/dispatcher.cc:344
#3 0x00007f33e362d0aa in Dispatcher::run (this=0x52ca700) at src/dispatcher.cc:186
#4 0x00007f33e362d86d in launch_dispatcher_thread (arg=0x52ca700) at src/dispatcher.cc:30
#5 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f33e8da390d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f33ea2f7720 (LWP 1368)):
#0 0x00007f33e8d67b8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f33e8d9cd64 in usleep () from /lib64/libc.so.6
#2 0x00007f33e365d91a in Flusher::wait (this=0x51b2a00) at src/flusher.cc:39
#3 0x00007f33e3630e0a in stopFlusher (this=0x513e300, __in_chrg=<value optimized out>)
at src/ep.cc:369
#4 EventuallyPersistentStore::~EventuallyPersistentStore (this=0x513e300,
__in_chrg=<value optimized out>) at src/ep.cc:327
#5 0x00007f33e3655275 in ~EventuallyPersistentEngine (handle=<value optimized out>,
force=<value optimized out>) at src/ep_engine.h:522
#6 EvpDestroy (handle=<value optimized out>, force=<value optimized out>)
at src/ep_engine.cc:135
#7 0x00007f33e6b0a425 in bucket_shutdown_engine (key=<value optimized out>,
nkey=<value optimized out>, val=0x50ddf10, nval=<value optimized out>,
args=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:1375
#8 0x00007f33e6b112bc in genhash_iter (h=0x8c0060,
iterfunc=0x7f33e6b0a3f0 <bucket_shutdown_engine>, arg=0x0)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/engines/bucket_engine/genhash.c:263
#9 0x00007f33e6b0b6be in bucket_destroy (handle=0x7f33e6d14480, force=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/engines/bucket_engine/bucket_engine.c:1412
#10 0x000000000040a3d8 in main (argc=<value optimized out>, argv=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/daemon/mem--Type <return> to continue, or q <return> to quit--
cached.c:7761

Thread 1 (Thread 0x7f33d6681700 (LWP 1401)):
#0 0x00007f33e8ced8a5 in raise () from /lib64/libc.so.6
#1 0x00007f33e8cef085 in abort () from /lib64/libc.so.6
#2 0x00007f33e9ada631 in cb_mutex_enter (mutex=<value optimized out>)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/platform/src/cb_pthreads.c:83
#3 0x00000000004150e6 in notify_io_complete (cookie=0x4f6c780, status=ENGINE_SUCCESS)
at /home/buildbot/cent54-master-toy-couchstore-builder/build/build/cmake/memcached/daemon/thread.c:467

#4 0x00007f33e36821aa in notifyIOComplete (this=<value optimized out>, tp=0xe134ec00)
at src/ep_engine.h:419
#5 TapConnMap::notifyPausedConnection (this=<value optimized out>, tp=0xe134ec00)
at src/tapconnmap.cc:439
#6 0x00007f33e361b552 in performTapOp<Item*> (this=0x1139ef290, gv=<value optimized out>)
at src/tapconnmap.h:168
#7 BackfillDiskCallback::callback (this=0x1139ef290, gv=<value optimized out>)
at src/backfill.cc:60
#8 0x00007f33e36a1efb in CouchKVStore::recordDbDump (db=0x86ff8b70, docinfo=0xe7f6ae00,
ctx=0x7f33d6680aa0) at src/couch-kvstore/couch-kvstore.cc:1308
#9 0x00007f33e33e859d in lookup_callback (rq=<value optimized out>, k=0x7f33d6680260,
v=0x7f33d6680250) at src/couch_db.cc:696

#10 0x00007f33e33e7294 in btree_lookup_inner (rq=0x7f33d6680510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:78
#11 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d6680510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#12 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d6680510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#13 0x00007f33e33e70f6 in btree_lookup_inner (rq=0x7f33d6680510, diskpos=<value optimized out>,
current=0, end=1) at src/btree_read.cc:52
#14 0x00007f33e33e97f6 in couchstore_changes_since (db=<value optimized out>,
--Type <return> to continue, or q <return> to quit--
since=<value optimized out>, options=<value optimized out>, callback=<value optimized out>,
ctx=<value optimized out>) at src/couch_db.cc:739
#15 0x00007f33e36a5d51 in CouchKVStore::loadDB (this=0x5187500,
cb=std::tr1::shared_ptr (count 5) 0x1139ef290, keysOnly=false, vbids=<value optimized out>,
options=0) at src/couch-kvstore/couch-kvstore.cc:952
#16 0x00007f33e36a6883 in CouchKVStore::dump (this=0x5187500, vb=184, cb=<value optimized out>)
at src/couch-kvstore/couch-kvstore.cc:767
#17 0x00007f33e361c675 in BackfillDiskLoad::run (this=0xf2e5fa40) at src/backfill.cc:79
#18 0x00007f33e3669d20 in ExecutorThread::run (this=0x50d1200) at src/scheduler.cc:97
#19 0x00007f33e366a1ad in launch_executor_thread (arg=<value optimized out>)
at src/scheduler.cc:38
#20 0x00007f33e9461851 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f33e8da390d in clone () from /lib64/libc.so.6