memcached gets stuck in shutdown code

Seeing this all the time, with quite high probability.

1. start cluster_run cluster

2. Ctrl-C, killing everything. memcached is supposed to shutdown itself. But sometimes I see memcached process(es) still alive.

Here's backtrace from recent such incident:

(gdb) thre app all bt

Thread 3 (Thread 0x7f24000ec700 (LWP 5203)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
#1 0x00007f2400707ce3 in logger_thead_main (arg=0x189b7b0) at extensions/loggers/file_logger.c:368
#2 0x00007f2401699b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#3 0x00007f24013e470d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f23f60d9700 (LWP 5213)):
#0 0x00007f24016a1950 in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f240169bf4d in __pthread_mutex_lock_full (mutex=0x599af40) at pthread_mutex_lock.c:316
#2 0x000000000040606e in release_cookie (cookie=0x592fe70) at daemon/memcached.c:6691
#3 0x00007f23feeaf636 in bucket_engine_release_cookie (cookie=0x592fe70) at bucket_engine.c:2565
#4 0x00007f23fc3ecf0a in EventuallyPersistentEngine::releaseCookie(void const*) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0x00007f23fc4163fe in TapConnection::releaseReference(bool) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#6 0x00007f23fc427eeb in TapConnectionReaperCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#7 0x00007f23fc3d3a5d in Dispatcher::run() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#8 0x00007f23fc3d52db in launch_dispatcher_thread () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#9 0x00007f2401699b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#10 0x00007f24013e470d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#11 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f24025b7700 (LWP 5200)):
#0 0x00007f240169ae75 in pthread_join (threadid=139792428668672, thread_return=0x0) at pthread_join.c:89
#1 0x00007f23fc3d0f21 in Dispatcher::stop(bool) () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#2 0x00007f23fc3db092 in EventuallyPersistentStore::~EventuallyPersistentStore() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#3 0x00007f23fc3ff38d in EventuallyPersistentEngine::~EventuallyPersistentEngine() () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#4 0x00007f23fc3f08b0 in EvpDestroy () from /root/src/altoros/moxi/repo20/install/lib/memcached/ep.so
#5 0x00007f23feeacd75 in bucket_shutdown_engine (key=<optimized out>, nkey=<optimized out>, val=0x599b810, nval=<optimized out>, args=<optimized out>)
at bucket_engine.c:1290
#6 0x00007f23feeb1ff4 in genhash_iter (h=0x5990bb0, iterfunc=iterfunc@entry=0x7f23feeacd40 <bucket_shutdown_engine>, arg=arg@entry=0x0) at genhash.c:275
#7 0x00007f23feeb0fea in bucket_destroy (handle=0x7f23ff0b49e0, force=<optimized out>) at bucket_engine.c:1327
#8 bucket_destroy (handle=0x7f23ff0b49e0, force=<optimized out>) at bucket_engine.c:1307
#9 0x0000000000404749 in main (argc=<optimized out>, argv=<optimized out>) at daemon/memcached.c:7921
(gdb)

apparently somebody 'leaked' locked mutex.