memcached crashed in EventuallyPersistentStore::flushVBucket -> Rebalance exited with reason {mover_failed,{badmatch,{error,closed}}

http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions-P1/219/consoleFull

./testrunner -i /tmp/rebalance_regression.ini wait_timeout=100,get-cbcollect-info=True -t swaprebalance.SwapRebalanceFailedTests.test_failover_swap_rebalance,replica=2,num-buckets=2,num-swap=2,keys-count=1000000,swap-orchestrator=True

crash on root/couchbase@10.3.121.94:

root@ubuntu1104-64:/tmp# ls -la
total 1446004
drwxrwxrwt 2 root root 4096 2013-06-10 00:17 .
drwxr-xr-x 21 root root 4096 2012-09-14 06:34 ..
rw------ 1 couchbase couchbase 1545568256 2013-06-08 14:55 core.memcached.701
rw-rr- 1 root root 124030928 2013-06-06 18:47 couchbase-server-enterprise_x86_64_2.1.0-708-rel.deb
root@ubuntu1104-64:/tmp# gdb /opt/couchbase/bin/memcached core.memcached.701
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 751]
[New Thread 752]
[New Thread 755]
[New Thread 792]
[New Thread 789]
[New Thread 706]
[New Thread 790]
[New Thread 756]
[New Thread 753]
[New Thread 703]
[New Thread 791]
[New Thread 750]
[New Thread 793]
[New Thread 701]
[New Thread 708]
[New Thread 702]
[New Thread 754]
[New Thread 707]
[New Thread 704]
[New Thread 705]
[New Thread 788]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x00007fec6b1f8d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) t a a bt

Thread 21 (Thread 788):
#0 0x00007fec6b2a643d in fdatasync () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec65f6eaaf in couch_sync (handle=<value optimized out>) at src/os.c:117
#2 0x00007fec66224eef in cfs_sync (h=0x4678c680) at src/couch-kvstore/couch-fs-stats.cc:88
#3 0x00007fec65f68e03 in couchstore_commit (db=0x6f044d0) at src/couch_db.c:184
#4 0x00007fec6621a3a6 in CouchKVStore::saveDocs (this=0x6f7d500, vbid=729, rev=<value optimized out>, docs=<value optimized out>, docinfos=0x49144320, docCount=<value optimized out>)
at src/couch-kvstore/couch-kvstore.cc:1485
#5 0x00007fec6621aafb in CouchKVStore::commit2couchstore (this=0x6f7d500) at src/couch-kvstore/couch-kvstore.cc:1410
#6 0x00007fec6621acea in CouchKVStore::commit (this=0x32) at src/couch-kvstore/couch-kvstore.cc:806
#7 0x00007fec661abff6 in EventuallyPersistentStore::flushVBucket (this=0x6f62800, vbid=<value optimized out>) at src/ep.cc:1963
#8 0x00007fec661d51d9 in doFlush (this=0x26a1540, tid=2745) at src/flusher.cc:222
#9 Flusher::step (this=0x26a1540, tid=2745) at src/flusher.cc:152
#10 0x00007fec661df379 in ExecutorThread::run (this=0x91ff040) at src/scheduler.cc:151
#11 0x00007fec661dfa3d in launch_executor_thread (arg=0x32) at src/scheduler.cc:35
#12 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x0000000000000000 in ?? ()

Thread 20 (Thread 705):
#0 0x00007fec6b2ae633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6c06cf36 in epoll_dispatch (base=0x6efe280, tv=<value optimized out>) at epoll.c:404
#2 0x00007fec6c058394 in event_base_loop (base=0x6efe280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414c84 in worker_libevent (arg=0x26994f8) at daemon/thread.c:301
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 19 (Thread 704):
#0 0x00007fec6b2ae633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6c06cf36 in epoll_dispatch (base=0x6efe500, tv=<value optimized out>) at epoll.c:404
#2 0x00007fec6c058394 in event_base_loop (base=0x6efe500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414c84 in worker_libevent (arg=0x2699400) at daemon/thread.c:301
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 18 (Thread 707):
#0 0x00007fec6b2ae633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6c06cf36 in epoll_dispatch (base=0x6efea00, tv=<value optimized out>) at epoll.c:404
#2 0x00007fec6c058394 in event_base_loop (base=0x6efea00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414c84 in worker_libevent (arg=0x26996e8) at daemon/thread.c:301
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 17 (Thread 754):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec661ba602 in wait (this=0x6f4e000) at src/syncobject.hh:57
#2 wait (this=0x6f4e000) at src/syncobject.hh:73
#3 wait (this=0x6f4e000) at src/tapconnmap.hh:171
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x6f4e000) at src/ep_engine.cc:3419
#5 0x00007fec661ba6f3 in EvpNotifyPendingConns (arg=0x6f4e000) at src/ep_engine.cc:1159
#6 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
--Type <return> to continue, or q <return> to quit--
#8 0x0000000000000000 in ?? ()

Thread 16 (Thread 702):
#0 0x00007fec6b29f2ed in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6b23a798 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fec6b23b7be in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fec6b22f8fa in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fec6b22e7ca in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fec6aaa6b19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#7 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 15 (Thread 708):
#0 0x00007fec6b2ae633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6c06cf36 in epoll_dispatch (base=0x6efe780, tv=<value optimized out>) at epoll.c:404
#2 0x00007fec6c058394 in event_base_loop (base=0x6efe780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414c84 in worker_libevent (arg=0x26997e0) at daemon/thread.c:301
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 14 (Thread 701):
#0 0x00007fec6b2ae633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6c06cf36 in epoll_dispatch (base=0x6efe000, tv=<value optimized out>) at epoll.c:404
#2 0x00007fec6c058394 in event_base_loop (base=0x6efe000, flags=<value optimized out>) at event.c:1558
#3 0x000000000040c841 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7926

Thread 13 (Thread 793):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec6619c3e8 in wait (this=0x7a89a70, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x7a89a70, d=...) at src/dispatcher.cc:342
#3 0x00007fec6619ef9a in Dispatcher::run (this=0x91d21c0) at src/dispatcher.cc:184
#4 0x00007fec6619f76d in launch_dispatcher_thread (arg=0x91d2214) at src/dispatcher.cc:28
#5 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 12 (Thread 750):
#0 0x00007fec6b2754ed in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6b2a6914 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fec661dc975 in updateStatsThread (arg=<value optimized out>) at src/memory_tracker.cc:31
#3 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 11 (Thread 791):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec661ba602 in wait (this=0x6f4f200) at src/syncobject.hh:57
#2 wait (this=0x6f4f200) at src/syncobject.hh:73
#3 wait (this=0x6f4f200) at src/tapconnmap.hh:171
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x6f4f200) at src/ep_engine.cc:3419
#5 0x00007fec661ba6f3 in EvpNotifyPendingConns (arg=0x6f4f200) at src/ep_engine.cc:1159
#6 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#7 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 10 (Thread 703):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec6a0a3176 in logger_thead_main (arg=<value optimized out>) at extensions/loggers/file_logger.c:368
#2 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()

Thread 9 (Thread 753):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec661df24e in wait (this=0x6fae680) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x6fae680) at src/scheduler.cc:141
#3 0x00007fec661dfa3d in launch_executor_thread (arg=0x6fae6c4) at src/scheduler.cc:35
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 8 (Thread 756):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec6619c3e8 in wait (this=0x269e360, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x269e360, d=...) at src/dispatcher.cc:342
#3 0x00007fec6619ef9a in Dispatcher::run (this=0x6f47880) at src/dispatcher.cc:184
#4 0x00007fec6619f76d in launch_dispatcher_thread (arg=0x6f478d4) at src/dispatcher.cc:28
#5 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 7 (Thread 790):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec661df24e in wait (this=0x91fed00) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x91fed00) at src/scheduler.cc:141
#3 0x00007fec661dfa3d in launch_executor_thread (arg=0x91fed44) at src/scheduler.cc:35
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 6 (Thread 706):
#0 0x00007fec6b2ae633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6c06cf36 in epoll_dispatch (base=0x6efec80, tv=<value optimized out>) at epoll.c:404
#2 0x00007fec6c058394 in event_base_loop (base=0x6efec80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414c84 in worker_libevent (arg=0x26995f0) at daemon/thread.c:301
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 789):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec661df24e in wait (this=0x91feea0) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x91feea0) at src/scheduler.cc:141
#3 0x00007fec661dfa3d in launch_executor_thread (arg=0x91feee4) at src/scheduler.cc:35
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()
--Type <return> to continue, or q <return> to quit--

Thread 4 (Thread 792):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec6619c3e8 in wait (this=0x7a88990, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x7a88990, d=...) at src/dispatcher.cc:342
#3 0x00007fec6619ef9a in Dispatcher::run (this=0x91d2380) at src/dispatcher.cc:184
#4 0x00007fec6619f76d in launch_dispatcher_thread (arg=0x91d23d4) at src/dispatcher.cc:28
#5 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 3 (Thread 755):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec6619c3e8 in wait (this=0x269e120, d=...) at src/syncobject.hh:57
#2 IdleTask::run (this=0x269e120, d=...) at src/dispatcher.cc:342
#3 0x00007fec6619ef9a in Dispatcher::run (this=0x6f46a80) at src/dispatcher.cc:184
#4 0x00007fec6619f76d in launch_dispatcher_thread (arg=0x6f46ad4) at src/dispatcher.cc:28
#5 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 2 (Thread 752):
#0 0x00007fec6b568f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fec661df24e in wait (this=0x6fae820) at src/syncobject.hh:57
#2 ExecutorThread::run (this=0x6fae820) at src/scheduler.cc:141
#3 0x00007fec661dfa3d in launch_executor_thread (arg=0x6fae864) at src/scheduler.cc:35
#4 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 751):
#0 0x00007fec6b1f8d05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fec6b1fcab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fec6b1f17c5 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fec661b7c86 in PersistenceCallback::callback(std::pair<int, long>&) () from /opt/couchbase/lib/memcached/ep.so
#4 0x00007fec66218fe4 in CouchKVStore::commitCallback(CouchRequest **, int, <anonymous enum>) (this=0x6f7c000, committedReqs=<value optimized out>, numReqs=1, errCode=COUCHSTORE_SUCCESS)
at src/couch-kvstore/couch-kvstore.cc:1590
#5 0x00007fec6621ab16 in CouchKVStore::commit2couchstore (this=0x6f7c000) at src/couch-kvstore/couch-kvstore.cc:1417
#6 0x00007fec6621acea in CouchKVStore::commit (this=0x2bd) at src/couch-kvstore/couch-kvstore.cc:806
#7 0x00007fec661abff6 in EventuallyPersistentStore::flushVBucket (this=0x6f62000, vbid=<value optimized out>) at src/ep.cc:1963
#8 0x00007fec661d51d9 in doFlush (this=0x26a1040, tid=4) at src/flusher.cc:222
#9 Flusher::step (this=0x26a1040, tid=4) at src/flusher.cc:152
#10 0x00007fec661df379 in ExecutorThread::run (this=0x6fae9c0) at src/scheduler.cc:151
#11 0x00007fec661dfa3d in launch_executor_thread (arg=0x2bd) at src/scheduler.cc:35
#12 0x00007fec6b563d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#13 0x00007fec6b2adfdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#14 0x0000000000000000 in ?? ()

test logs:

[2013-06-08 14:54:44,126] - [rest_client:1014] ERROR -

{u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

- rebalance failed
[2013-06-08 14:54:44,126] - [rest_client:1015] INFO - Latest logs from UI:
[2013-06-08 14:54:44,128] - [mcsoda:1216] ERROR - errorcode = 134
[2013-06-08 14:54:44,129] - [mcsoda:1216] ERROR - errorcode = 134
[2013-06-08 14:54:44,129] - [mcsoda:1234] INFO - inflight recv backoff = 0.8
[2013-06-08 14:54:44,259] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 1, u'text': u"Node 'ns_1@10.3.121.93' is leaving cluster.", u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1370728509413, u'type': u'info'}

[2013-06-08 14:54:44,260] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 4, u'text': u'Node ns_1@10.3.121.93 left cluster', u'shortText': u'message', u'module': u'ns_cluster', u'tstamp': 1370728509331, u'type': u'info'}

[2013-06-08 14:54:44,261] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 2, u'text': u'Rebalance exited with reason {mover_failed,{badmatch,

{error,closed}}}\n', u'shortText': u'message', u'module': u'ns_orchestrator', u'tstamp': 1370728509153, u'type': u'info'}
[2013-06-08 14:54:44,261] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'<0.17378.96> exited with {mover_failed,{badmatch,{error,closed}

}}', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370728509121, u'type': u'critical'}
[2013-06-08 14:54:44,262] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.94', u'code': 0, u'text': u'Port server memcached on node \'babysitter_of_ns_1@127.0.0.1\' exited with status 134. Restarting. Messages: Sat Jun 8 14:54:56.490676 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_990_\'ns_1@10.3.121.95\' - Connection is closed by force\nSat Jun 8 14:54:56.490652 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:56.490981 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:56.491155 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 662\nSat Jun 8 14:54:56.498791 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2843"\nSat Jun 8 14:54:56.498970 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_990_\'ns_1@10.3.121.95\' - Clear the tap queues by force\nSat Jun 8 14:54:56.555072 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Schedule the backfill for vbucket 662\nSat Jun 8 14:54:56.555120 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:56.555133 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:56.555142 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 662\nSat Jun 8 14:54:56.614081 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Backfill is completed with VBuckets 662, \nSat Jun 8 14:54:56.614208 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 662\nSat Jun 8 14:54:56.667988 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:56.668193 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:56.668288 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - Sending TAP_VBUCKET_SET with vbucket 990 and state "pending"\nSat Jun 8 14:54:56.670478 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Backfill is completed with VBuckets 662, \nSat Jun 8 14:54:56.670660 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 662\nSat Jun 8 14:54:56.678457 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - VBucket <990> is going dead to complete vbucket takeover\nSat Jun 8 14:54:56.702476 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - Sending TAP_VBUCKET_SET with vbucket 990 and state "active"\nSat Jun 8 14:54:56.716452 PDT 3: (bucket-1) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_990>\nSat Jun 8 14:54:56.716618 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - disconnected\nSat Jun 8 14:54:56.717067 PDT 3: (bucket-1) Clean up "eq_tapq:rebalance_990"\nSat Jun 8 14:54:56.717102 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_990 - Clear the tap queues by force\nSat Jun 8 14:54:56.758898 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_990_\'ns_1@10.3.121.96\' - disconnected, keep alive for 300 seconds\nSat Jun 8 14:54:56.766061 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_990_\'ns_1@10.3.121.96\' - Connection is closed by force\nSat Jun 8 14:54:56.820832 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2844"\nSat Jun 8 14:54:56.820985 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_990_\'ns_1@10.3.121.96\' - Clear the tap queues by force\nSat Jun 8 14:54:56.882805 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Schedule the backfill for vbucket 989\nSat Jun 8 14:54:56.883567 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:56.883654 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:56.883667 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 989\nSat Jun 8 14:54:56.917426 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - disconnected, keep alive for 300 seconds\nSat Jun 8 14:54:56.937481 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Schedule the backfill for vbucket 989\nSat Jun 8 14:54:56.938300 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:56.938339 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:56.938353 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 989\nSat Jun 8 14:54:56.945494 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Connection is closed by force\nSat Jun 8 14:54:56.956164 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2845"\nSat Jun 8 14:54:56.956200 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.96\' - Clear the tap queues by force\nSat Jun 8 14:54:57.031385 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Backfill is completed with VBuckets 989, \nSat Jun 8 14:54:57.031434 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 989\nSat Jun 8 14:54:57.064133 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Backfill is completed with VBuckets 989, \nSat Jun 8 14:54:57.064266 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 989\nSat Jun 8 14:54:57.155368 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:57.155399 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:57.155410 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - Sending TAP_VBUCKET_SET with vbucket 662 and state "pending"\nSat Jun 8 14:54:57.178139 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - VBucket <662> is going dead to complete vbucket takeover\nSat Jun 8 14:54:57.189843 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - Sending TAP_VBUCKET_SET with vbucket 662 and state "active"\nSat Jun 8 14:54:57.194184 PDT 3: (bucket-1) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_662>\nSat Jun 8 14:54:57.194272 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - disconnected\nSat Jun 8 14:54:57.194677 PDT 3: (bucket-1) Clean up "eq_tapq:rebalance_662"\nSat Jun 8 14:54:57.194711 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_662 - Clear the tap queues by force\nSat Jun 8 14:54:57.217379 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - disconnected, keep alive for 300 seconds\nSat Jun 8 14:54:57.225820 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Connection is closed by force\nSat Jun 8 14:54:57.302236 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - disconnected, keep alive for 300 seconds\nSat Jun 8 14:54:57.313913 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Connection is closed by force\nSat Jun 8 14:54:57.353711 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.96\' - Schedule the backfill for vbucket 988\nSat Jun 8 14:54:57.353863 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:57.353959 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:57.354052 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 988\nSat Jun 8 14:54:57.354424 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2846"\nSat Jun 8 14:54:57.354459 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_662_\'ns_1@10.3.121.95\' - Clear the tap queues by force\nSat Jun 8 14:54:57.354469 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2847"\nSat Jun 8 14:54:57.362258 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Schedule the backfill for vbucket 988\nSat Jun 8 14:54:57.362397 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:57.362411 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:57.362420 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 988\nSat Jun 8 14:54:57.379100 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.95\' - Clear the tap queues by force\nSat Jun 8 14:54:57.484953 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:57.485084 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:57.486121 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - Sending TAP_VBUCKET_SET with vbucket 989 and state "pending"\nSat Jun 8 14:54:57.496162 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - VBucket <989> is going dead to complete vbucket takeover\nSat Jun 8 14:54:57.502229 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.96\' - Backfill is completed with VBuckets 988, \nSat Jun 8 14:54:57.502270 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 988\nSat Jun 8 14:54:57.506470 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - Sending TAP_VBUCKET_SET with vbucket 989 and state "active"\nSat Jun 8 14:54:57.514470 PDT 3: (bucket-1) TAP takeover is completed. Disconnecting tap stream <eq_tapq:rebalance_989>\nSat Jun 8 14:54:57.514683 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - disconnected\nSat Jun 8 14:54:57.515064 PDT 3: (bucket-1) Clean up "eq_tapq:rebalance_989"\nSat Jun 8 14:54:57.515079 PDT 3: (bucket-1) TAP (Producer) eq_tapq:rebalance_989 - Clear the tap queues by force\nSat Jun 8 14:54:57.541274 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - disconnected, keep alive for 300 seconds\nSat Jun 8 14:54:57.542447 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Backfill is completed with VBuckets 988, \nSat Jun 8 14:54:57.542580 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 988\nSat Jun 8 14:54:57.551337 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Connection is closed by force\nSat Jun 8 14:54:57.558071 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2848"\nSat Jun 8 14:54:57.558128 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_989_\'ns_1@10.3.121.96\' - Clear the tap queues by force\nSat Jun 8 14:54:57.817881 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.96\' - Schedule the backfill for vbucket 661\nSat Jun 8 14:54:57.818287 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:57.818399 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:57.818517 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 661\nSat Jun 8 14:54:57.881369 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - disconnected, keep alive for 300 seconds\nSat Jun 8 14:54:57.890067 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Connection is closed by force\nSat Jun 8 14:54:57.895554 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.96\' - Backfill is completed with VBuckets 661, \nSat Jun 8 14:54:57.895851 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.96\' - Sending TAP_OPAQUE with command "close_backfill" and vbucket 661\nSat Jun 8 14:54:57.895962 PDT 3: (bucket-1) Clean up "eq_tapq:anon_2849"\nSat Jun 8 14:54:57.896018 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_988_\'ns_1@10.3.121.95\' - Clear the tap queues by force\nSat Jun 8 14:54:57.934127 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.95\' - Schedule the backfill for vbucket 661\nSat Jun 8 14:54:57.934267 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "opaque_enable_auto_nack" and vbucket 0\nSat Jun 8 14:54:57.934364 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "enable_checkpoint_sync" and vbucket 0\nSat Jun 8 14:54:57.934459 PDT 3: (bucket-1) TAP (Producer) eq_tapq:replication_building_661_\'ns_1@10.3.121.95\' - Sending TAP_OPAQUE with command "initial_vbucket_stream" and vbucket 661\nmemcached: src/ep.cc:1777: virtual void PersistenceCallback::callback(mutation_result&): Assertion `stats->diskQueueSize < ((size_t)1<<(sizeof(size_t)*8-1))\' failed.', u'shortText': u'message', u'module': u'ns_log', u'tstamp': 1370728502183, u'type': u'info'}

[2013-06-08 14:54:44,265] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.94', u'code': 4, u'text': u"Control connection to memcached on 'ns_1@10.3.121.94' disconnected: {badmatch,\n {error,\n closed}}", u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370728502182, u'type': u'info'}
[2013-06-08 14:54:44,266] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.93', u'code': 0, u'text': u'Bucket "bucket-1" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'module': u'ns_vbucket_mover', u'tstamp': 1370728464850, u'type': u'info'}

[2013-06-08 14:54:44,267] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.94', u'code': 5, u'text': u"Node 'ns_1@10.3.121.94' saw that node 'ns_1@10.3.121.98' went down. Details: [

{nodedown_reason,\n connection_closed}]", u'shortText': u'node down', u'module': u'ns_node_disco', u'tstamp': 1370728463744, u'type': u'warning'}
[2013-06-08 14:54:44,268] - [rest_client:1016] ERROR - {u'node': u'ns_1@10.3.121.93', u'code': 5, u'text': u"Node 'ns_1@10.3.121.93' saw that node 'ns_1@10.3.121.98' went down. Details: [{nodedown_reason,n connection_closed}

]", u'shortText': u'node down', u'module': u'ns_node_disco', u'tstamp': 1370728463712, u'type': u'warning'}
[2013-06-08 14:54:44,268] - [rest_client:1016] ERROR -

{u'node': u'ns_1@10.3.121.95', u'code': 1, u'text': u'Bucket "bucket-1" loaded on node \'ns_1@10.3.121.95\' in 0 seconds.', u'shortText': u'message', u'module': u'ns_memcached', u'tstamp': 1370728463703, u'type': u'info'}