memcached hang when create a second bucket after resize first bucket

Install couchbase server 3.0.0-513 on 2 nodes cluster
10.3.121.17
10.6.2.139
Copy ep.so from build 3.0.0-403 into these 2 nodes cluster to do IBR test.
Create default bucket with all RAM available (1 replica, 2.9 GB)
Load 100K items into default bucket
Resize default bucket down to 1GB
Create a second sasl bucket. Memcached hang
Cluster also hang

Backtrace from memcached

Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install couchbase-server-3.0.0-513.x86_64
(gdb) thread apply all bt

Thread 17 (Thread 0x7f6f86e45700 (LWP 8168)):
#0 0x00007f6f889f560d in read () from /lib64/libc.so.6
#1 0x00007f6f8898bf68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f6f8898da6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f6f8898214a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f6f88980fa9 in fgets () from /lib64/libc.so.6
#5 0x00007f6f86e468c1 in check_stdin_thread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890070) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#7 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f6f86241700 (LWP 8169)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0x7f6f86444800, mutex=0x7f6f864447c0, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f86243d4d in logger_thead_main (arg=0x28d0b00) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/extensions/loggers/file_logger.c:362
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890080) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f6f85634700 (LWP 8181)):
#0 0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f6f89b25376 in epoll_dispatch (base=0xbc1c500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f6f89b10c44 in event_base_loop (base=0xbc1c500, flags=<value optimized out>) at event.c:1558
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890180) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f6f84c33700 (LWP 8182)):
#0 0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f6f89b25376 in epoll_dispatch (base=0xbc1c280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f6f89b10c44 in event_base_loop (base=0xbc1c280, flags=<value optimized out>) at event.c:1558
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890170) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f6f84232700 (LWP 8183)):
#0 0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f6f89b25376 in epoll_dispatch (base=0xbc1cc80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f6f89b10c44 in event_base_loop (base=0xbc1cc80, flags=<value optimized out>) at event.c:1558
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890160) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f6f83831700 (LWP 8184)):
#0 0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f6f89b25376 in epoll_dispatch (base=0xbc1ca00, tv=<value optimized out>) at epoll.c:404
#2 0x00007f6f89b10c44 in event_base_loop (base=0xbc1ca00, flags=<value optimized out>) at event.c:1558
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890150) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f6f82e30700 (LWP 8185)):
#0 0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f6f89b25376 in epoll_dispatch (base=0xbc1c780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f6f89b10c44 in event_base_loop (base=0xbc1c780, flags=<value optimized out>) at event.c:1558
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890140) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f6f7fdd4700 (LWP 8186)):
--Type <return> to continue, or q <return> to quit--
#0 0x00007f6f889c6b8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f6f889fbd64 in usleep () from /lib64/libc.so.6
#2 0x00007f6f8216b055 in updateStatsThread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/memory_tracker.cc:36
#3 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x28901b0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#4 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f6f7f3d3700 (LWP 8187)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4ba in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=75 'K') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:128
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f46e0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890c10) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f6f7e9d2700 (LWP 8188)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4d5 in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=252 '\374') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:144
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f4690) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890bf0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f6f7dfd1700 (LWP 8189)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4d5 in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=41 ')') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:144
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f4640) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890c00) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f6f7d5d0700 (LWP 8190)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4d5 in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=52 '4') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:144
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f45f0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890be0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f6f7cbcf700 (LWP 8191)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4d5 in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=154 '\232') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:144
--Type <return> to continue, or q <return> to quit--
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f49b0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890bd0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f6f7c1ce700 (LWP 8192)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4ba in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=205 '\315') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:128
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f4960) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890bc0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f6f7b7cd700 (LWP 8193)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4d5 in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=91 '[') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:144
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f4910) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890bb0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f6f7adcc700 (LWP 8194)):
#0 0x00007f6f893487bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f6f8a38c8fb in cb_cond_timedwait (cond=0xbc78de0, mutex=0xbc78da8, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:156
#2 0x00007f6f8215e287 in wait (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0xbc78d80, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:170
#4 0x00007f6f8215e4d5 in ExecutorPool::nextTask (this=0xbc78d80, t=..., tick=223 '\337') at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorpool.cc:144
#5 0x00007f6f8216d229 in ExecutorThread::run (this=0x28f48c0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:77
#6 0x00007f6f8216d7b6 in launch_executor_thread (arg=0xbc78de4) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/ep-engine/src/executorthread.cc:33
#7 0x00007f6f8a38cb7f in platform_thread_wrap (arg=0x2890ba0) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/platform/src/cb_pthreads.c:19
#8 0x00007f6f89344851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f6f88a0290d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f6f8aba47e0 (LWP 8156)):
#0 0x00007f6f88a02f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f6f89b25376 in epoll_dispatch (base=0xbc1c000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f6f89b10c44 in event_base_loop (base=0xbc1c000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040dbeb in main (argc=<value optimized out>, argv=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/cmake/memcached/daemon/memcached.c:8284
(gdb)