Crash in executor pool unlockShardTask function

Note that I have also seen a similar unit test failure on ubuntu systems. It seems sporadic and happens to random test cases at different times.

Another stack from the simple.conf , which looks quite different.

• Note this is build 3.0.0-749-rel, it still does not contain the above fix.
• Attaching core files https://s3.amazonaws.com/bugdb/11217/core.memcached.31791

[root@cherry-s10126 tmp]# gdb /opt/couchbase/bin/memcached core.memcached.31791
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 2277]
[New Thread 31793]
[New Thread 31798]
[New Thread 31791]
[New Thread 31795]
[New Thread 31792]
[New Thread 31799]
[New Thread 31796]
[New Thread 2266]
[New Thread 31794]
[New Thread 31797]
[New Thread 32189]
[New Thread 2343]
[New Thread 2279]
[New Thread 2280]
[New Thread 2278]
Reading symbols from /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/memcached/libmcd_util.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcbsasl.so.1.1.1
Reading symbols from /opt/couchbase/bin/../lib/libplatform.so.0.1.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libplatform.so.0.1.0
Reading symbols from /opt/couchbase/bin/../lib/libcJSON.so.1.0.0...done.
Loaded symbols for /opt/couchbase/bin/../lib/libcJSON.so.1.0.0
Reading symbols from /opt/couchbase/bin/../lib/libJSON_checker.so...done.
Loaded symbols for /opt/couchbase/bin/../lib/libJSON_checker.so
Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1
Reading symbols from /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/bin/../lib/libtcmalloc_minimal.so.4
Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.0.so.5...done.
Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.0.so.5
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so
Reading symbols from /opt/couchbase/lib/libdirutils.so.0.1.0...done.
Loaded symbols for /opt/couchbase/lib/libdirutils.so.0.1.0
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 6, Aborted.
#0 0x00007fcf032048a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-749.x86_64
(gdb) t a a bt

Thread 16 (Thread 0x7fcef81af700 (LWP 2278)):
#0 0x00007fcf032b385d in fdatasync () from /lib64/libc.so.6
#1 0x00007fcefcca2357 in couch_sync (errinfo=0x898f020, handle=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/os.c:133
#2 0x00007fcefcf9519b in cfs_sync(struct

{...} *, couch_file_handle) (errinfo=0x898f020, h=0x9d115e0)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3 0x00007fcefcc8c6ea in couchstore_commit (db=0x898f000)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:203
#4 0x00007fcefcf87149 in CouchKVStore::setVBucketState (this=0x6f95400, vbucketId=533, vbstate=..., vb_change_type=2, kvcb=
0x7fcef81aec60, notify=true) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:962
#5 0x00007fcefcf878d0 in CouchKVStore::snapshotVBuckets (this=0x6f95400, vbstates=std::map with 256 elements = {...}

, cb=0x7fcef81aec60)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:846
#6 0x00007fcefceddb6f in EventuallyPersistentStore::snapshotVBuckets (this=0x6ddf840, priority=..., shardId=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:853
#7 0x00007fcefcf48e1f in VBSnapshotTask::run (this=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:44
#8 0x00007fcefcf245a1 in ExecutorThread::run (this=0x8dfb360)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#9 0x00007fcefcf24ab6 in launch_executor_thread (arg=0x40)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#10 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506ba0)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fcef9fb2700 (LWP 2280)):
#0 0x00007fcf0411fec3 in pwrite64 () from /lib64/libpthread.so.0
#1 0x00007fcefcca2431 in couch_pwrite (errinfo=0x6fbf8a0, handle=<value optimized out>, buf=0x8f08031, nbyte=91, offset=24576)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/os.c:65
#2 0x00007fcefcf94fc1 in cfs_pwrite(struct

{...} *, couch_file_handle, const void *, size_t, cs_off_t) (errinfo=0x6fbf8a0, h=0x6e558c0,
buf=0x8f08031, sz=91, off=24576)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:115
#3 0x00007fcefcc909d1 in flush_buffer (errinfo=0x6fbf8a0, handle=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/iobuffer.cc:113
#4 buffered_sync(struct {...}

*, couch_file_handle) (errinfo=0x6fbf8a0, handle=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/iobuffer.cc:394
#5 0x00007fcefcc8c6ea in couchstore_commit (db=0x6fbf880)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:203
#6 0x00007fcefcf87149 in CouchKVStore::setVBucketState (this=0x6f94700, vbucketId=403, vbstate=..., vb_change_type=2,
kvcb=0x7fcef9fb1c60, notify=true)
--Type <return> to continue, or q <return> to quit--
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:962
#7 0x00007fcefcf878d0 in CouchKVStore::snapshotVBuckets (this=0x6f94700, vbstates=std::map with 256 elements =

{...}, cb=0x7fcef9fb1c60)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:846
#8 0x00007fcefceddb6f in EventuallyPersistentStore::snapshotVBuckets (this=0x6ddf840, priority=..., shardId=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:853
#9 0x00007fcefcf48e1f in VBSnapshotTask::run (this=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:44
#10 0x00007fcefcf245a1 in ExecutorThread::run (this=0x8dfae60)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#11 0x00007fcefcf24ab6 in launch_executor_thread (arg=0x42)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#12 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506b80)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#13 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#14 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fcef8bb0700 (LWP 2279)):
#0 0x00007fcf032b385d in fdatasync () from /lib64/libc.so.6
#1 0x00007fcefcca2357 in couch_sync (errinfo=0x8978320, handle=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/os.c:133
#2 0x00007fcefcf9519b in cfs_sync(struct {...}

*, couch_file_handle) (errinfo=0x8978320, h=0x70fb4a0)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-fs-stats.cc:128
#3 0x00007fcefcc8c6ea in couchstore_commit (db=0x8978300)
at /home/buildbot/centos-5-x64-300-builder/build/build/couchstore/src/couch_db.cc:203
#4 0x00007fcefcf87149 in CouchKVStore::setVBucketState (this=0x6f943c0, vbucketId=514, vbstate=..., vb_change_type=2,
kvcb=0x7fcef8bafc60, notify=true)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:962
#5 0x00007fcefcf878d0 in CouchKVStore::snapshotVBuckets (this=0x6f943c0, vbstates=std::map with 256 elements =

{...}

, cb=0x7fcef8bafc60)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:846
#6 0x00007fcefceddb6f in EventuallyPersistentStore::snapshotVBuckets (this=0x6ddf840, priority=..., shardId=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:853
#7 0x00007fcefcf48e1f in VBSnapshotTask::run (this=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:44
#8 0x00007fcefcf245a1 in ExecutorThread::run (this=0x8dfc7b0)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#9 0x00007fcefcf24ab6 in launch_executor_thread (arg=0x41)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#10 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506d80)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fcf032ba90d in clone () from /lib64/libc.so.6
--Type <return> to continue, or q <return> to quit--

Thread 13 (Thread 0x7fcef63ac700 (LWP 2343)):
#0 0x00007fcf0411f054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fcf0411a388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007fcf0411a257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fcf053629f9 in cb_mutex_enter (mutex=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007fcefcf234dd in Mutex::acquire (this=0x6e15dc0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/mutex.cc:31
#5 0x00007fcefcf4d71e in lock (this=0x6e15dc0, task=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/locks.h:66
#6 LockHolder (this=0x6e15dc0, task=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/locks.h:44
#7 TaskQueue::wake (this=0x6e15dc0, task=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:174
#8 0x00007fcefcf13ee4 in ExecutorPool::stopTaskGroup(EventuallyPersistentEngine *, ._252) (this=0x6e14d80, e=0x8ea8a00,
taskType=NO_TASK_TYPE) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:456
#9 0x00007fcefcf14176 in ExecutorPool::unregisterBucket (this=0x6e14d80, engine=0x8ea8a00)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:476
#10 0x00007fcefced62e7 in EventuallyPersistentStore::~EventuallyPersistentStore (this=0x6ddf840, __in_chrg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:361
#11 0x00007fcefcef2895 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x8ea8a00, __in_chrg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:5509
#12 0x00007fcefcef2b25 in EvpDestroy (handle=0x8ea8a00, force=true)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep_engine.cc:143
#13 0x00007fcf003fa2fb in engine_shutdown_thread (arg=0x7a702a0)
at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1554
#14 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2507250)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#15 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fcef95b1700 (LWP 32189)):
#0 0x00007fcf0411c43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fcf003fa414 in engine_shutdown_thread (arg=0x6dd0620)
at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1600
#2 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506fa0)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fcefe5f3700 (LWP 31797)):
#0 0x00007fcf032baf03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fcf048f9376 in epoll_dispatch (base=0x6dbea00, tv=<value optimized out>) at epoll.c:404
#2 0x00007fcf048e4c44 in event_base_loop (base=0x6dbea00, flags=<value optimized out>) at event.c:1558
--Type <return> to continue, or q <return> to quit--
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506170)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fcf003f6700 (LWP 31794)):
#0 0x00007fcf032baf03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fcf048f9376 in epoll_dispatch (base=0x6dbe500, tv=<value optimized out>) at epoll.c:404
#2 0x00007fcf048e4c44 in event_base_loop (base=0x6dbe500, flags=<value optimized out>) at event.c:1558
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x25061a0)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fcef6dad700 (LWP 2266)):
#0 0x00007fcf0411c43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fcf003fa414 in engine_shutdown_thread (arg=0x8543490)
at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/engines/bucket_engine/bucket_engine.c:1600
#2 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506d60)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#3 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fcefeff4700 (LWP 31796)):
#0 0x00007fcf032baf03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fcf048f9376 in epoll_dispatch (base=0x6dbec80, tv=<value optimized out>) at epoll.c:404
#2 0x00007fcf048e4c44 in event_base_loop (base=0x6dbec80, flags=<value optimized out>) at event.c:1558
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506180)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fcefa9b3700 (LWP 31799)):
#0 0x00007fcf0327eb8d in nanosleep () from /lib64/libc.so.6
#1 0x00007fcf032b3d64 in usleep () from /lib64/libc.so.6
#2 0x00007fcefcf220b5 in updateStatsThread (arg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x25061f0)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6
--Type <return> to continue, or q <return> to quit--

Thread 6 (Thread 0x7fcf01c19700 (LWP 31792)):
#0 0x00007fcf032ad60d in read () from /lib64/libc.so.6
#1 0x00007fcf03243f68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007fcf03245a6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007fcf0323a14a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007fcf03238fa9 in fgets () from /lib64/libc.so.6
#5 0x00007fcf01c1a8b1 in check_stdin_thread (arg=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506080)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#7 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fceff9f5700 (LWP 31795)):
#0 0x00007fcf032baf03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fcf048f9376 in epoll_dispatch (base=0x6dbe280, tv=<value optimized out>) at epoll.c:404
#2 0x00007fcf048e4c44 in event_base_loop (base=0x6dbe280, flags=<value optimized out>) at event.c:1558
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506190)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fcf05b7a7e0 (LWP 31791)):
#0 0x00007fcf032baf03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fcf048f9376 in epoll_dispatch (base=0x6dbe000, tv=<value optimized out>) at epoll.c:404
#2 0x00007fcf048e4c44 in event_base_loop (base=0x6dbe000, flags=<value optimized out>) at event.c:1558

#3 0x000000000040f169 in main (argc=<value optimized out>, argv=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8786

Thread 3 (Thread 0x7fcefdbf2700 (LWP 31798)):
#0 0x00007fcf032baf03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007fcf048f9376 in epoll_dispatch (base=0x6dbe780, tv=<value optimized out>) at epoll.c:404
#2 0x00007fcf048e4c44 in event_base_loop (base=0x6dbe780, flags=<value optimized out>) at event.c:1558
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506160)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fcf01004700 (LWP 31793)):
#0 0x00007fcf0411c7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#1 0x00007fcf053628eb in cb_cond_timedwait (cond=0x7fcf01218e60, mutex=0x7fcf01218e20, ms=<value optimized out>)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007fcf01008548 in logger_thead_main (arg=0x253eb00)
at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007fcf05362b6f in platform_thread_wrap (arg=0x2506090)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fcf032ba90d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fcef77ae700 (LWP 2277)):
#0 0x00007fcf032048a5 in raise () from /lib64/libc.so.6
#1 0x00007fcf03206085 in abort () from /lib64/libc.so.6
#2 0x00007fcf031fda1e in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007fcf031fdae0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007fcefcf4bf77 in unlockShardTask (this=0x6e15dc0, task=<value optimized out>, curTaskType=@0x8dfa958)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/kvshard.h:168
#5 unlockShard (this=0x6e15dc0, task=<value optimized out>, curTaskType=@0x8dfa958)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.h:566
#6 TaskQueue::doneTask_UNLOCKED(ExTask &, ._241 &) (this=0x6e15dc0, task=<value optimized out>, curTaskType=@0x8dfa958)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:145
#7 0x00007fcefcf4c1ff in TaskQueue::doneTask(ExTask &, ._241 &) (this=0x6e15dc0, task=..., curTaskType=@0x8dfa958)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/taskqueue.cc:160
#8 0x00007fcefcf244b0 in ExecutorThread::run (this=0x8dfa910)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:81
#9 0x00007fcefcf24ab6 in launch_executor_thread (arg=0x7c2f)
at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#10 0x00007fcf05362b6f in platform_thread_wrap (arg=0x25061e0)
at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#11 0x00007fcf04118851 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fcf032ba90d in clone () from /lib64/libc.so.6