memcached on master node crashed (Segmentation fault) when failover other nodes

Install coucbase server 3.0.0-829 on 12 nodes centos 6.4 64-bit
Create cluster with 4 zones, each zone has 3 nodes
1:10.3.4.225
2:10.3.4.227
3:10.3.4.228

4:10.3.4.229
5:10.3.4.230
6:10.3.4.231

7:10.3.4.234
8:10.3.4.233
9:10.3.4.232

10:10.3.4.235
11:10.1.2.86
12:10.1.2.84

Create 2 buckets with 3 replica, default and standard_bucket0
Load 100K items into each bucket
Failover all nodes in zone 2, 3 and 4
Memcached crashed (segfault) on master node 10.3.4.225

Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
Program terminated with signal 11, Segmentation fault.
#0 SLL_Pop (size=35, nothrow=false) at src/linked_list.h:59
59 src/linked_list.h: No such file or directory.
in src/linked_list.h

Thread 13 (Thread 0x7f88dc6cd700 (LWP 16313)):
#0 0x00007f88e7c3a7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f88e8e828eb in cb_cond_timedwait (cond=0x5899940, mutex=0x5899908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f88e002f7d7 in wait (this=0x58998e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x58998e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007f88e002fa25 in ExecutorPool::nextTask (this=0x58998e0, t=..., tick=8 '\b') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007f88e0040e5d in ExecutorThread::run (this=0xfd6480) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007f88e0041416 in launch_executor_thread (arg=0x5899944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82bb0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f88dd0ce700 (LWP 16312)):
#0 0x00007f88e7c3a7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f88e8e828eb in cb_cond_timedwait (cond=0x5899940, mutex=0x5899908, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f88e002f7d7 in wait (this=0x58998e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/syncobject.h:74
#3 ExecutorPool::trySleep (this=0x58998e0, t=..., now=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:177
#4 0x00007f88e002fa25 in ExecutorPool::nextTask (this=0x58998e0, t=..., tick=24 '\030') at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorpool.cc:151
#5 0x00007f88e0040e5d in ExecutorThread::run (this=0xfd63f0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:77
#6 0x00007f88e0041416 in launch_executor_thread (arg=0x5899944) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#7 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82bc0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f88e0d0f700 (LWP 16310)):
#0 0x00007f88e6d9cb8d in nanosleep () from /lib64/libc.so.6
#1 0x00007f88e6dd1d64 in usleep () from /lib64/libc.so.6
#2 0x00007f88e003e6e5 in updateStatsThread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/memory_tracker.cc:36
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf821f0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f88e2111700 (LWP 16210)):
#0 0x00007f88e6dd8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f88e8418376 in epoll_dispatch (base=0x5842a00, tv=<value optimized out>) at epoll.c:404
#2 0x00007f88e8403c44 in event_base_loop (base=0x5842a00, flags=<value optimized out>) at event.c:1558
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82130) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f88e2b12700 (LWP 16209)):
#0 0x00007f88e6dd8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f88e8418376 in epoll_dispatch (base=0x5842780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f88e8403c44 in event_base_loop (base=0x5842780, flags=<value optimized out>) at event.c:1558
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82120) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f88e4b22700 (LWP 16206)):
#0 0x00007f88e7c3a7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f88e8e828eb in cb_cond_timedwait (cond=0x7f88e4d36e60, mutex=0x7f88e4d36e20, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f88e4b26548 in logger_thead_main (arg=0xfc2900) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82080) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f88dbccc700 (LWP 16314)):
#0 0x00007f88e7c3d054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f88e7c38388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f88e7c38257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f88e8e829f9 in cb_mutex_enter (mutex=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:85
#4 0x00007f88e003fb0d in Mutex::acquire (this=0x58b8110) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/mutex.cc:31
#5 0x00007f88e00b511f in lock (this=0x58b8000, vb=128, cb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/locks.h:66
#6 LockHolder (this=0x58b8000, vb=128, cb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/locks.h:44
#7 CouchNotifier::delVBucket (this=0x58b8000, vb=128, cb=...) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-notifier.cc:777
#8 0x00007f88e00a78a5 in CouchKVStore::delVBucket (this=0x58ae700, vbucket=163, recreate=true) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:532
#9 0x00007f88dffef779 in EventuallyPersistentStore::completeVBucketDeletion (this=0x587a240, vbid=<value optimized out>, cookie=0x0, recreate=true) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:974
#10 0x00007f88e0065ad9 in VBDeleteTask::run (this=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:50
#11 0x00007f88e0040ef1 in ExecutorThread::run (this=0xfd6510) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#12 0x00007f88e0041416 in launch_executor_thread (arg=0x58b8118) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#13 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82bd0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#14 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f88e3f14700 (LWP 16207)):
#0 0x00007f88e6dd8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f88e8418376 in epoll_dispatch (base=0x5842280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f88e8403c44 in event_base_loop (base=0x5842280, flags=<value optimized out>) at event.c:1558
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82100) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f88e1710700 (LWP 16211)):
#0 0x00007f88e6dd8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f88e8418376 in epoll_dispatch (base=0x5842c80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f88e8403c44 in event_base_loop (base=0x5842c80, flags=<value optimized out>) at event.c:1558
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82140) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f88e5737700 (LWP 16205)):
#0 0x00007f88e6dcb60d in read () from /lib64/libc.so.6
#1 0x00007f88e6d61f68 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f88e6d63a6e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f88e6d5814a in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007f88e6d56fa9 in fgets () from /lib64/libc.so.6
#5 0x00007f88e57388b1 in check_stdin_thread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf820e0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#7 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f88e3513700 (LWP 16208)):
#0 0x00007f88e6dd8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f88e8418376 in epoll_dispatch (base=0x5842500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f88e8403c44 in event_base_loop (base=0x5842500, flags=<value optimized out>) at event.c:1558
#3 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82110) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f88e6dd890d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f88e969a7e0 (LWP 16204)):
#0 0x00007f88e6dd8f03 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f88e8418376 in epoll_dispatch (base=0x5842000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f88e8403c44 in event_base_loop (base=0x5842000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040f1c9 in main (argc=<value optimized out>, argv=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/daemon/memcached.c:8794

Thread 1 (Thread 0x7f88ddacf700 (LWP 16311)):
#0 SLL_Pop (size=35, nothrow=false) at src/linked_list.h:59
#1 Pop (size=35, nothrow=false) at src/thread_cache.h:216
#2 Allocate (size=35, nothrow=false) at src/thread_cache.h:368
#3 do_malloc_small (size=35, nothrow=false) at src/tcmalloc.cc:1103
#4 do_malloc_no_errno (size=35, nothrow=false) at src/tcmalloc.cc:1110
#5 (anonymous namespace)::cpp_alloc (size=35, nothrow=false) at src/tcmalloc.cc:1438
#6 0x00007f88e86408a8 in tc_new (size=35) at src/tcmalloc.cc:1616
#7 0x00007f88e75b93c9 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6
#8 0x00007f88e75b9fb5 in ?? () from /usr/lib64/libstdc++.so.6
#9 0x00007f88e75ba119 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long) () from /u
sr/lib64/libstdc++.so.6
#10 0x00007f88dfb9d350 in substr (input="/opt/couchbase/var/lib/couchbase/data/default/120.couch.", directory=true) at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/basic_string.h:1908
#11 CouchbaseDirectoryUtilities::split (input="/opt/couchbase/var/lib/couchbase/data/default/120.couch.", directory=true) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/dirutils.cc:57
#12 0x00007f88dfb9d67e in CouchbaseDirectoryUtilities::dirname (dir=Traceback (most recent call last):
File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 556, in to_string
header = ptr.cast(reptype) - 1
RuntimeError: Cannot access memory at address 0x4
) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/dirutils.cc:70
#13 0x00007f88dfb9dcb6 in CouchbaseDirectoryUtilities::findFilesWithPrefix (name="/opt/couchbase/var/lib/couchbase/data/default/120.couch.") at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/dirutils.cc:136
#14 0x00007f88e009df9a in CouchKVStore::checkNewRevNum (this=<value optimized out>, dbFileName="/opt/couchbase/var/lib/couchbase/data/default/120.couch.4", newFile=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/
build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:1325
#15 0x00007f88e009e59d in CouchKVStore::openDB (this=0x58ac000, vbucketId=<value optimized out>, fileRev=4, db=0x7f88ddace9c8, options=1, newFileRev=0x7f88ddace9c0) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/
src/couch-kvstore/couch-kvstore.cc:1376
#16 0x00007f88e009f882 in CouchKVStore::setVBucketState (this=0x58ac000, vbucketId=120, vbstate=..., vb_change_type=1, kvcb=0x0, notify=true) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch
-kvstore.cc:1024
#17 0x00007f88e00a7b66 in resetVBucket (this=0x58ac000, vbucket=120, recreate=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.h:577
#18 CouchKVStore::delVBucket (this=0x58ac000, vbucket=120, recreate=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/couch-kvstore/couch-kvstore.cc:542
#19 0x00007f88dffef779 in EventuallyPersistentStore::completeVBucketDeletion (this=0x587a240, vbid=<value optimized out>, cookie=0x0, recreate=true) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/ep.cc:974
#20 0x00007f88e0065ad9 in VBDeleteTask::run (this=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/tasks.cc:50
#21 0x00007f88e0040ef1 in ExecutorThread::run (this=0xfd6360) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:94
#22 0x00007f88e0041416 in launch_executor_thread (arg=0x23) at /home/buildbot/centos-5-x64-300-builder/build/build/ep-engine/src/executorthread.cc:33
#23 0x00007f88e8e82b6f in platform_thread_wrap (arg=0xf82ba0) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#24 0x00007f88e7c36851 in start_thread () from /lib64/libpthread.so.0
#25 0x00007f88e6dd890d in clone () from /lib64/libc.so.6
-------------------