sigabort in memcached during rebalance after failover

Lots of incoming connections to memcached, 3 couchbase nodes with 600 moxi clients connecting.

Create a 3 node cluster with just the default bucket of 100MB

Started up 600 client side moxis (2 nodes, 300 moxis for each)

Started a small load vs both moxis, the load does go into DGM

During this load I rebalanced in 3 new nodes, then rebalanced them out.

The abort happened on one of the nodes being rebalanced in and out.

Thread 12 (Thread 0x2b1aa6c43220 (LWP 21949)):
#0 0x0000003e820d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b1aa67c3c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5
#2 0x00002b1aa67b2a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5
#3 0x0000000000409994 in pthread_attr_setdetachstate ()
#4 0x0000003e8201d994 in __libc_start_main () from /lib64/libc.so.6
#5 0x0000000000402b39 in pthread_attr_setdetachstate ()
#6 0x00007fff89565458 in ?? ()
#7 0x0000000000000000 in ?? ()

Thread 11 (Thread 21958):
#0 0x0000003e820c678b in read () from /lib64/libc.so.6
#1 0x0000003e8206cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003e8206d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003e82068fdb in getc () from /lib64/libc.so.6
#4 0x00002b1aa6c44875 in ?? () from /opt/couchbase/lib/memcached/stdin_term_handler.so
#5 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 21959):
#0 0x0000003e8209a541 in nanosleep () from /lib64/libc.so.6
#1 0x0000003e8209a364 in sleep () from /lib64/libc.so.6
#2 0x0000000000414f78 in pthread_attr_setdetachstate ()
#3 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 9 (Thread 21960):
#0 0x0000003e820d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b1aa67c3c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5
#2 0x00002b1aa67b2a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5
#3 0x00000000004133b4 in pthread_attr_setdetachstate ()
#4 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 21962):
#0 0x0000003e820d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b1aa67c3c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5
#2 0x00002b1aa67b2a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5
#3 0x00000000004133b4 in pthread_attr_setdetachstate ()
#4 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 21963):
#0 0x0000003e820d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b1aa67c3c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5
#2 0x00002b1aa67b2a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5
#3 0x00000000004133b4 in pthread_attr_setdetachstate ()
#4 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 21964):
#0 0x0000003e820d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002b1aa67c3c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5
#2 0x00002b1aa67b2a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5
#3 0x00000000004133b4 in pthread_attr_setdetachstate ()
#4 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 21965):
#0 0x0000003e82c0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaad07b43 in Dispatcher::run() () from /opt/couchbase/lib/memcached/ep.so
#2 0x00002aaaaad08753 in ?? () from /opt/couchbase/lib/memcached/ep.so
#3 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 21966):
#0 0x0000003e820cda97 in fdatasync () from /lib64/libc.so.6
#1 0x00002aaaaad84799 in ?? () from /opt/couchbase/lib/memcached/ep.so
#2 0x00002aaaaad9f7c8 in ?? () from /opt/couchbase/lib/memcached/ep.so
#3 0x00002aaaaadce9b7 in sqlite3_wal_checkpoint () from /opt/couchbase/lib/memcached/ep.so
#4 0x00002aaaaadceb71 in ?? () from /opt/couchbase/lib/memcached/ep.so
#5 0x00002aaaaae0a487 in sqlite3_step () from /opt/couchbase/lib/memcached/ep.so
#6 0x00002aaaaad784c9 in PreparedStatement::execute() () from /opt/couchbase/lib/memcached/ep.so
#7 0x00002aaaaad77c0b in StrategicSqlite3::commit() () from /opt/couchbase/lib/memcached/ep.so
#8 0x00002aaaaad0b46e in TransactionContext::commit() () from /opt/couchbase/lib/memcached/ep.so
#9 0x00002aaaaad14766 in EventuallyPersistentStore::flushSome(std::queue<RCPtr<QueuedItem>, std::deque<RCPtr<QueuedItem>, std::allocator<RCPtr<QueuedItem> > > >, std::queue<RCPtr<QueuedItem>, std::deque<RCPtr<QueuedItem>, std::allocator<RCPtr<QueuedItem> > > >) () from /opt/couchbase/lib/memcached/ep.so
#10 0x00002aaaaad48255 in Flusher::doFlush() () from /opt/couchbase/lib/memcached/ep.so
#11 0x00002aaaaad48a15 in Flusher::step(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#12 0x00002aaaaad491be in FlusherStepper::callback(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#13 0x00002aaaaad0914f in Task::run(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#14 0x00002aaaaad07d1f in Dispatcher::run() () from /opt/couchbase/lib/memcached/ep.so
#15 0x00002aaaaad08753 in ?? () from /opt/couchbase/lib/memcached/ep.so
#16 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#17 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 3 (Thread 21967):
#0 0x0000003e82c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00002aaaaad059d0 in IdleTask::run(Dispatcher&, std::tr1::shared_ptr<Task>) () from /opt/couchbase/lib/memcached/ep.so
#2 0x00002aaaaad07d1f in Dispatcher::run() () from /opt/couchbase/lib/memcached/ep.so
#3 0x00002aaaaad08753 in ?? () from /opt/couchbase/lib/memcached/ep.so
#4 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 21978):
#0 0x0000003e8209a541 in nanosleep () from /lib64/libc.so.6
#1 0x0000003e820cded4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaad46f0a in Flusher::wait() () from /opt/couchbase/lib/memcached/ep.so
#3 0x00002aaaaad0f100 in EventuallyPersistentStore::~EventuallyPersistentStore() () from /opt/couchbase/lib/memcached/ep.so
#4 0x00002aaaaad42945 in EventuallyPersistentEngine::~EventuallyPersistentEngine() () from /opt/couchbase/lib/memcached/ep.so
#5 0x00002aaaaad31459 in ?? () from /opt/couchbase/lib/memcached/ep.so
#6 0x00002aaaaaaae8ef in ?? () from /opt/couchbase/lib/memcached/bucket_engine.so
#7 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003e820d44bd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x42c58940 (LWP 21961)):
#0 0x0000003e82030265 in raise () from /lib64/libc.so.6
#1 0x0000003e82031d10 in abort () from /lib64/libc.so.6
#2 0x0000003e820296e6 in __assert_fail () from /lib64/libc.so.6
#3 0x00002aaaaaaae092 in ?? () from /opt/couchbase/lib/memcached/bucket_engine.so
#4 0x00002aaaaaaae59c in ?? () from /opt/couchbase/lib/memcached/bucket_engine.so
#5 0x000000000040c22e in pthread_attr_setdetachstate ()
#6 0x00000000004108bb in pthread_attr_setdetachstate ()
#7 0x00000000004117b1 in pthread_attr_setdetachstate ()
#8 0x0000000000407414 in pthread_attr_setdetachstate ()
#9 0x00002b1aa67b2df9 in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5
#10 0x00000000004133b4 in pthread_attr_setdetachstate ()
#11 0x0000003e82c0673d in start_thread () from /lib64/libpthread.so.0
#12 0x0000003e820d44bd in clone () from /lib64/libc.so.6