[root@centos-64-x64 data]# gdb /opt/couchbase/bin/memcached core.memcached.14936 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 15797]
[New Thread 14943]
[New Thread 14975]
[New Thread 14942]
[New Thread 14936]
[New Thread 14973]
[New Thread 14974]
[New Thread 14978]
[New Thread 15795]
[New Thread 15447]
[New Thread 14979]
[New Thread 14977]
[New Thread 15590]
[New Thread 15601]
[New Thread 15793]
[New Thread 14976]
[New Thread 15794]
[New Thread 15796]
[New Thread 15792]
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 6, Aborted.
#0  0x00007f07aa83d8a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-883.x86_64
(gdb) t a a bt

Thread 19 (Thread 0x7f079cf82700 (LWP 15792)):
#0  0x00007f07aa8ec85d in fdatasync () from /lib64/libc.so.6
#1  0x00007f07a24d7637 in couch_sync (errinfo=0x84365a0, handle=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/os.c:133
#2  0x00007f07a27caf8b in cfs_sync(struct {...} *, couch_file_handle) (errinfo=0x84365a0, h=0xf7c99a0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3  0x00007f07a24c172a in couchstore_commit (db=0x8436580) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:203
#4  0x00007f07a27bc0b9 in CouchKVStore::setVBucketState (this=0x8804340, vbucketId=169, vbstate=..., vb_change_type=1, kvcb=0x7f079cf81c60, notify=true)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1045
#5  0x00007f07a27bc84f in CouchKVStore::snapshotVBuckets (this=0x8804340, vbstates=std::map with 64 elements = {...}, cb=0x7f079cf81c60)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:929
#6  0x00007f07a2719ecf in EventuallyPersistentStore::snapshotVBuckets (this=0x6619d40, priority=..., shardId=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:851
#7  0x00007f07a277e94f in VBSnapshotTask::run (this=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:44
#8  0x00007f07a275a081 in ExecutorThread::run (this=0x798cc70) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#9  0x00007f07a275a5a6 in launch_executor_thread (arg=0x66) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#10 0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f079f786700 (LWP 15796)):
#0  0x00007f07aa8ec85d in fdatasync () from /lib64/libc.so.6
#1  0x00007f07a24d7637 in couch_sync (errinfo=0x10adc120, handle=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/os.c:133
#2  0x00007f07a27caf8b in cfs_sync(struct {...} *, couch_file_handle) (errinfo=0x10adc120, h=0x10370440)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3  0x00007f07a24c172a in couchstore_commit (db=0x10adc100) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:203
#4  0x00007f07a27bada0 in CouchKVStore::saveDocs (this=0x7fcc000, vbid=110, rev=<value optimized out>, docs=0x1ceae78, docinfos=0x1ceaa10, docCount=<value optimized out>, kvctx=...)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1816
#5  0x00007f07a27bb806 in CouchKVStore::commit2couchstore (this=0x7fcc000, cb=0x7f079f7859a0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1710
#6  0x00007f07a27bbb25 in CouchKVStore::commit (this=0x77, cb=0x77) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1135
#7  0x00007f07a2718c8f in EventuallyPersistentStore::flushVBucket (this=0x661ad00, vbid=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:2509
#8  0x00007f07a274fccc in Flusher::flushVB (this=0x66466c0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/flusher.cc:281
#9  0x00007f07a275093e in Flusher::step (this=0x77, task=0x6e26780) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/flusher.cc:172
#10 0x00007f07a277e976 in FlusherTask::run (this=0x0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:40
#11 0x00007f07a275a081 in ExecutorThread::run (this=0x1d3a510) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#12 0x00007f07a275a5a6 in launch_executor_thread (arg=0x77) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#13 0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f079e384700 (LWP 15794)):
#0  0x00007f07aa8ec85d in fdatasync () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x00007f07a24d7637 in couch_sync (errinfo=0x8434c20, handle=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/os.c:133
#2  0x00007f07a27caf8b in cfs_sync(struct {...} *, couch_file_handle) (errinfo=0x8434c20, h=0xa91f280)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3  0x00007f07a24c16fa in couchstore_commit (db=0x8434c00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:194
#4  0x00007f07a27bc0b9 in CouchKVStore::setVBucketState (this=0x7fcc340, vbucketId=113, vbstate=..., vb_change_type=2, kvcb=0x7f079e383c60, notify=true)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1045
#5  0x00007f07a27bc84f in CouchKVStore::snapshotVBuckets (this=0x7fcc340, vbstates=std::map with 62 elements = {...}, cb=0x7f079e383c60)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:929
#6  0x00007f07a2719ecf in EventuallyPersistentStore::snapshotVBuckets (this=0x661ad00, priority=..., shardId=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:851
#7  0x00007f07a277e94f in VBSnapshotTask::run (this=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:44
#8  0x00007f07a275a081 in ExecutorThread::run (this=0x8b20120) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#9  0x00007f07a275a5a6 in launch_executor_thread (arg=0x79) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#10 0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f07a5c2c700 (LWP 14976)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a6a00, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a6a00, flags=<value optimized out>) at event.c:1558
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f079d983700 (LWP 15793)):
#0  snappy::internal::CompressFragment (input=0xde1a000 "\001\001 ", input_size=<value optimized out>, op=0x87ff2c7 "", table=0x7f079d971db0, table_size=<value optimized out>)
    at snappy.cc:390
#1  0x00007f07ac38f1f8 in snappy::Compress (reader=0x7f079d972620, writer=0x7f079d972640) at snappy.cc:864
#2  0x00007f07ac38f5ac in snappy::RawCompress (input=<value optimized out>, input_length=<value optimized out>, compressed=<value optimized out>, compressed_length=0x7f079d972688)
    at snappy.cc:1011
#3  0x00007f07a24c27de in db_write_buf_compressed (file=0x743c4a80, buf=0x7f079d9826e0, pos=0x7f079d982700, disk_size=0x7f079d9826f8)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_file_write.cc:137
#4  0x00007f07a24be069 in flush_mr_partial (res=0x38abc080, mr_quota=64) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/btree_modify.cc:244
#5  0x00007f07a24c82a2 in TreeWriterWrite (writer=0xdacee40, treefile=<value optimized out>, out_root=0x743c4ab8)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/tree_writer.cc:178
#6  0x00007f07a24c36f2 in couchstore_compact_db_ex(Db *, const char *, couchstore_compact_flags, couchstore_compact_hook, couchstore_docinfo_hook, void *, const struct {...} *) (
    source=0x743c4e00, target_filename=0x66800598 "/data/default/592.couch.13.compact", flags=0, hook=<value optimized out>, 
    dhook=0x7f07a27b2d90 <edit_docinfo_hook(DocInfo**, sized_buf const*)>, hook_ctx=0x10fe6cb8, ops=0x7f07a26de0e0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/db_compact.cc:64
#7  0x00007f07a27bd82e in CouchKVStore::compactVBucket(uint16_t, struct {...} *, Callback<compaction_ctx> &, Callback<KVStatsCtx> &) (this=0x89c2a40, vbid=592, hook_ctx=0x10fe6cb8, 
    cb=..., kvcb=...) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:799
#8  0x00007f07a2708555 in EventuallyPersistentStore::compactVBucket(uint16_t, struct {...} *, const void *) (this=0x6619d40, vbid=<value optimized out>, ctx=0x10fe6cb8, 
    cookie=0x645a600) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:1101
---Type <return> to continue, or q <return> to quit---
#9  0x00007f07a277ea89 in CompactVBucketTask::run (this=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:54
#10 0x00007f07a275a081 in ExecutorThread::run (this=0x798d0f0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#11 0x00007f07a275a5a6 in launch_executor_thread (arg=0xde1a26b) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#12 0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f079bb80700 (LWP 15601)):
#0  0x00007f07ab75543c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f07a7a33444 in engine_shutdown_thread (arg=0x65bc460) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1610
#2  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f079c581700 (LWP 15590)):
#0  0x00007f07ab75543c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f07a7a33444 in engine_shutdown_thread (arg=0x65bc690) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1610
#2  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f07a522b700 (LWP 14977)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a6c80, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a6c80, flags=<value optimized out>) at event.c:1558
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f07a3e29700 (LWP 14979)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a7180, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a7180, flags=<value optimized out>) at event.c:1558
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f07a3428700 (LWP 15447)):
#0  0x00007f07aa8b7b8d in nanosleep () from /lib64/libc.so.6
#1  0x00007f07aa8ecd64 in usleep () from /lib64/libc.so.6
#2  0x00007f07a2757875 in updateStatsThread (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 9 (Thread 0x7f07a0187700 (LWP 15795)):
#0  0x00007f07ab758ec3 in pwrite64 () from /lib64/libpthread.so.0
#1  0x00007f07a24d7711 in couch_pwrite (errinfo=0x7b35720, handle=<value optimized out>, buf=0x4efcc031, nbyte=1511, offset=3852941)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/os.c:65
#2  0x00007f07a27cadb1 in cfs_pwrite(struct {...} *, couch_file_handle, const void *, size_t, cs_off_t) (errinfo=0x7b35720, h=0xd1157c0, buf=0x4efcc031, sz=1511, off=3852941)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:115
#3  0x00007f07a24c60dd in flush_buffer (errinfo=0x7b35720, handle=0x74531ec0, buf=0x7f07a01864d0, nbyte=8, offset=3854534)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/iobuffer.cc:113
#4  buffered_pwrite(struct {...} *, couch_file_handle, const void *, size_t, cs_off_t) (errinfo=0x7b35720, handle=0x74531ec0, buf=0x7f07a01864d0, nbyte=8, offset=3854534)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/iobuffer.cc:356
#5  0x00007f07a24c263b in raw_write (file=0x7b35700, buf=0x7f07a0186530, pos=0x0, disk_size=0x0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_file_write.cc:40
#6  db_write_buf (file=0x7b35700, buf=0x7f07a0186530, pos=0x0, disk_size=0x0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_file_write.cc:101
#7  0x00007f07a24c16ec in couchstore_commit (db=0x7b35700) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:191
#8  0x00007f07a27bada0 in CouchKVStore::saveDocs (this=0x7f1c000, vbid=899, rev=<value optimized out>, docs=0x1ceafe8, docinfos=0x1ceafd8, docCount=<value optimized out>, kvctx=...)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1816
#9  0x00007f07a27bb806 in CouchKVStore::commit2couchstore (this=0x7f1c000, cb=0x7f07a01869a0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1710
#10 0x00007f07a27bbb25 in CouchKVStore::commit (this=0x72, cb=0x4efcc031)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1135
#11 0x00007f07a2718c8f in EventuallyPersistentStore::flushVBucket (this=0x6619d40, vbid=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:2509
#12 0x00007f07a274fccc in Flusher::flushVB (this=0x6645180) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/flusher.cc:281
#13 0x00007f07a275093e in Flusher::step (this=0x72, task=0x6e26fa0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/flusher.cc:172
#14 0x00007f07a277e976 in FlusherTask::run (this=0x573) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:40
#15 0x00007f07a275a081 in ExecutorThread::run (this=0x1d3a480) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#16 0x00007f07a275a5a6 in launch_executor_thread (arg=0x72) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#17 0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#18 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f07a482a700 (LWP 14978)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a6f00, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a6f00, flags=<value optimized out>) at event.c:1558
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f07a702e700 (LWP 14974)):
#0  0x00007f07ab758054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f07ab753388 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007f07ab753257 in pthread_mutex_lock () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#3  0x00007f07ac99da89 in cb_mutex_enter (mutex=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4  0x00007f07a2758c9d in Mutex::acquire (this=0x931d560) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/mutex.cc:31
#5  0x00007f07a278e8de in lock (this=0x931d500) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/locks.h:66
#6  LockHolder (this=0x931d500) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/locks.h:44
#7  PassiveStream::next (this=0x931d500) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:1114
#8  0x00007f07a27846bb in UprConsumer::getNextItem (this=0x81b5c00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:584
#9  0x00007f07a278488a in UprConsumer::step (this=0x81b5c00, producers=0x6222a0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:374
#10 0x00007f07a2724ad2 in EvpUprStep (handle=<value optimized out>, cookie=0x64a9300, producers=0x6222a0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:1350
#11 0x00007f07a7a381d6 in upr_step (handle=<value optimized out>, cookie=0x64a9300, producers=0x6222a0)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:2225
#12 0x000000000041347f in ship_upr_log (c=0x64a9300) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:3480
#13 conn_ship_log (c=0x64a9300) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:6833
#14 0x000000000040a37d in event_handler (fd=<value optimized out>, which=<value optimized out>, arg=0x64a9300)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:7276
#15 0x00007f07abf1ed3c in event_process_active_single_queue (base=0x65a6500, flags=<value optimized out>) at event.c:1308
#16 event_process_active (base=0x65a6500, flags=<value optimized out>) at event.c:1375
#17 event_base_loop (base=0x65a6500, flags=<value optimized out>) at event.c:1572
#18 0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#19 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f07a7a2f700 (LWP 14973)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a6280, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a6280, flags=<value optimized out>) at event.c:1558
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f07ad1b57e0 (LWP 14936)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a6000, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a6000, flags=<value optimized out>) at event.c:1558
#3  0x000000000040f71c in main (argc=<value optimized out>, argv=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8768

Thread 4 (Thread 0x7f07a9252700 (LWP 14942)):
#0  0x00007f07aa8e660d in read () from /lib64/libc.so.6
#1  0x00007f07aa87cf68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2  0x00007f07aa87ea6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3  0x00007f07aa87314a in _IO_getline_info_internal () from /lib64/libc.so.6
#4  0x00007f07aa871fa9 in fgets () from /lib64/libc.so.6
#5  0x00007f07a92538b1 in check_stdin_thread (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
---Type <return> to continue, or q <return> to quit---
#7  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f07a662d700 (LWP 14975)):
#0  0x00007f07aa8f3f03 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f07abf33376 in epoll_dispatch (base=0x65a6780, tv=<value optimized out>) at epoll.c:404
#2  0x00007f07abf1ec44 in event_base_loop (base=0x65a6780, flags=<value optimized out>) at event.c:1558
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f07a863d700 (LWP 14943)):
#0  0x00007f07ab7557bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f07ac99d97b in cb_cond_timedwait (cond=0x7f07a8851e60, mutex=0x7f07a8851e20, ms=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2  0x00007f07a8641548 in logger_thead_main (arg=0x1d26900) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4  0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f07aa8f390d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f079ed85700 (LWP 15797)):
#0  0x00007f07aa83d8a5 in raise () from /lib64/libc.so.6
#1  0x00007f07aa83f085 in abort () from /lib64/libc.so.6
#2  0x00007f07aa836a1e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007f07aa836ae0 in __assert_fail () from /lib64/libc.so.6
#4  0x00007f07a2794cc2 in PassiveStream::~PassiveStream (this=0x931d500, __in_chrg=<value optimized out>)
    at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/upr-stream.cc:852
#5  0x00007f07a2786485 in UprConsumer::processBufferedItems (this=0x81b5c00) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/atomic.h:397
#6  0x00007f07a278653a in Processer::run (this=0x7136eb0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/upr-consumer.cc:48
#7  0x00007f07a275a081 in ExecutorThread::run (this=0x1d3a5a0) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#8  0x00007f07a275a5a6 in launch_executor_thread (arg=0x3a58) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#9  0x00007f07ac99dbff in platform_thread_wrap (arg=<value optimized out>) at /buildbot/build_slave/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#10 0x00007f07ab751851 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f07aa8f390d in clone () from /lib64/libc.so.6