items not draining seems... ep-engine is deadlocked

during xdcr longevity test memcached kept restarting on a node
ui reports: Control connection to memcached on 'ns_1@172.23.105.57' disconnected: {badmatch,
{error,
couldnt_connect_to_memcached}}

(guinep-s1050.sc.couchbase.com)

everytime I do a gdb backtrace I see threads 16,17,18 in the lock() method.

Thread 18 (Thread 0x7f1212849700 (LWP 21113)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x641e0f0) at src/mutex.cc:79
#4 0x00007f121447d566 in lock (this=0x641e000) at ./src/locks.hh:48
#5 LockHolder (this=0x641e000) at ./src/locks.hh:26
#6 CouchNotifier::selectBucket (this=0x641e000) at src/couch-kvstore/couch-notifier.cc:723
#7 0x00007f121447dbcf in CouchNotifier::processInput (this=0x641e000) at src/couch-kvstore/couch-notifier.cc:606
#8 0x00007f121447e475 in waitOnce (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at src/couch-kvstore/couch-notifier.cc:675
#9 CouchNotifier::notify_update (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00007f1214475cb8 in notify_headerpos_update (this=0x643fb00, vbid=760, rev=1, docs=0x323bd880, docinfos=0x323bda40, docCount=56) at ./src/couch-kvstore/couch-notifier.hh:144
#11 CouchKVStore::saveDocs (this=0x643fb00, vbid=760, rev=1, docs=0x323bd880, docinfos=0x323bda40, docCount=56) at src/couch-kvstore/couch-kvstore.cc:1498
#12 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x643fb00) at src/couch-kvstore/couch-kvstore.cc:1410
#13 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#14 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x6395c00, vbid=760) at src/ep.cc:1919
#15 0x00007f121442aeb9 in doFlush (this=0x520b7a0, tid=1098) at src/flusher.cc:222
#16 Flusher::step (this=0x520b7a0, tid=1098) at src/flusher.cc:152
#17 0x00007f121443ac10 in ExecutorThread::run (this=0x52bfba0) at src/scheduler.cc:148
#18 0x00007f121443b32d in launch_executor_thread (arg=0x52bfba0) at src/scheduler.cc:34
#19 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f1211e48700 (LWP 21114)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x641e0f0) at src/mutex.cc:79
#4 0x00007f121447e2c3 in lock (this=0x641e000, vbs=..., file_version=1, header_offset=217088, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x641e000, vbs=..., file_version=1, header_offset=217088, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x641e000, vbs=..., file_version=1, header_offset=217088, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00007f1214475cb8 in notify_headerpos_update (this=0x64d4c00, vbid=693, rev=1, docs=0x93f4480, docinfos=0x94186c0, docCount=72) at ./src/couch-kvstore/couch-notifier.hh:144
#8 CouchKVStore::saveDocs (this=0x64d4c00, vbid=693, rev=1, docs=0x93f4480, docinfos=0x94186c0, docCount=72) at src/couch-kvstore/couch-kvstore.cc:1498
#9 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x64d4c00) at src/couch-kvstore/couch-kvstore.cc:1410
#10 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#11 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x6395c00, vbid=693) at src/ep.cc:1919
#12 0x00007f121442aeb9 in doFlush (this=0x520b680, tid=1095) at src/flusher.cc:222
#13 Flusher::step (this=0x520b680, tid=1095) at src/flusher.cc:152
#14 0x00007f121443ac10 in ExecutorThread::run (this=0x52bfa00) at src/scheduler.cc:148
#15 0x00007f121443b32d in launch_executor_thread (arg=0x52bfa00) at src/scheduler.cc:34
#16 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f1211447700 (LWP 21115)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x641e0f0) at src/mutex.cc:79
#4 0x00007f121447e2c3 in lock (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at ./src/locks.hh:48
#5 LockHolder (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at ./src/locks.hh:26
#6 CouchNotifier::notify_update (this=0x641e000, vbs=..., file_version=1, header_offset=212992, cb=...) at src/couch-kvstore/couch-notifier.cc:753
#7 0x00007f1214475cb8 in notify_headerpos_update (this=0x64d4600, vbid=750, rev=1, docs=0x321a4480, docinfos=0x321a4fc0, docCount=65) at ./src/couch-kvstore/couch-notifier.hh:144
#8 CouchKVStore::saveDocs (this=0x64d4600, vbid=750, rev=1, docs=0x321a4480, docinfos=0x321a4fc0, docCount=65) at src/couch-kvstore/couch-kvstore.cc:1498
#9 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x64d4600) at src/couch-kvstore/couch-kvstore.cc:1410
#10 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#11 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x6395c00, vbid=750) at src/ep.cc:1919
#12 0x00007f121442aeb9 in doFlush (this=0x520b560, tid=1096) at src/flusher.cc:222
#13 Flusher::step (this=0x520b560, tid=1096) at src/flusher.cc:152
#14 0x00007f121443ac10 in ExecutorThread::run (this=0x52bf860) at src/scheduler.cc:148
#15 0x00007f121443b32d in launch_executor_thread (arg=0x52bf860) at src/scheduler.cc:34
#16 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f1210a46700 (LWP 21116)):
#0 0x00007f121a5b4054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f121a5af388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f121a5af257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f1214436f7a in Mutex::acquire (this=0x528f0f0) at src/mutex.cc:79
#4 0x00007f121447d566 in lock (this=0x528f000) at ./src/locks.hh:48
#5 LockHolder (this=0x528f000) at ./src/locks.hh:26
#6 CouchNotifier::selectBucket (this=0x528f000) at src/couch-kvstore/couch-notifier.cc:723
#7 0x00007f121447dbcf in CouchNotifier::processInput (this=0x528f000) at src/couch-kvstore/couch-notifier.cc:606
#8 0x00007f121447e475 in waitOnce (this=0x528f000, vbs=..., file_version=2, header_offset=1798144, cb=...) at src/couch-kvstore/couch-notifier.cc:675
#9 CouchNotifier::notify_update (this=0x528f000, vbs=..., file_version=2, header_offset=1798144, cb=...) at src/couch-kvstore/couch-notifier.cc:755
#10 0x00007f1214475cb8 in notify_headerpos_update (this=0x5295b00, vbid=1023, rev=2, docs=0x1a897c00, docinfos=0x1a896a80, docCount=107) at ./src/couch-kvstore/couch-notifier.hh:144
#11 CouchKVStore::saveDocs (this=0x5295b00, vbid=1023, rev=2, docs=0x1a897c00, docinfos=0x1a896a80, docCount=107) at src/couch-kvstore/couch-kvstore.cc:1498
#12 0x00007f121447628b in CouchKVStore::commit2couchstore (this=0x5295b00) at src/couch-kvstore/couch-kvstore.cc:1410
#13 0x00007f121447647a in CouchKVStore::commit (this=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:806
#14 0x00007f1214401f56 in EventuallyPersistentStore::flushVBucket (this=0x527e000, vbid=1023) at src/ep.cc:1919
#15 0x00007f121442aeb9 in doFlush (this=0x520a360, tid=12) at src/flusher.cc:222
#16 Flusher::step (this=0x520a360, tid=12) at src/flusher.cc:152
#17 0x00007f121443ac10 in ExecutorThread::run (this=0x52e6ea0) at src/scheduler.cc:148
#18 0x00007f121443b32d in launch_executor_thread (arg=0x52e6ea0) at src/scheduler.cc:34
#19 0x00007f121a5ad851 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f121a2fb90d in clone () from /lib64/libc.so.6

there are also errors in memcached.logs about "Too many connections"

I have full backtrace attached and machine is currently live.