memcached crashed in flush_buffer -> Rebalance exited with reason {mover_failed,{badmatch,{error,closed}}}

2.0.0-1787-rel
http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/57/console
./testrunner -i /tmp/rebalance_regression.ini get-logs=True,disabled_consistent_view=False -t rebalancetests.IncrementalRebalanceInTests.test_load,replica=1,delete-ratio=0.6,expiry-ratio=0.2,keys-count=1000000

(disabled_consistent_view=False)

gdb /opt/couchbase/bin/memcached core.memcached.28393
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New Thread 28410]
[New Thread 28411]
[New Thread 28413]
[New Thread 28393]
[New Thread 28401]
[New Thread 28414]
[New Thread 28412]
[New Thread 28405]
[New Thread 28402]
[New Thread 28406]
[New Thread 28404]
[New Thread 28408]
[New Thread 28407]
[New Thread 28409]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Reading symbols from /opt/couchbase/lib/libcouchstore.so.1...done.
Loaded symbols for /opt/couchbase/lib/libcouchstore.so.1
Reading symbols from /opt/couchbase/lib/libsnappy.so.1...done.
Loaded symbols for /opt/couchbase/lib/libsnappy.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f2d34a116db in flush_buffer (handle=<value optimized out>, buf=0x7f2d33df2490, nbyte=8, offset=180315) at src/iobuffer.c:106
106 src/iobuffer.c: No such file or directory.
in src/iobuffer.c
(gdb) t a a bt

Thread 14 (Thread 28409):
#0 0x00007f2d3950d4ed in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3953e914 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f2d34c7c525 in updateStatsThread (arg=<value optimized out>) at src/memory_tracker.cc:31
#3 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()

Thread 13 (Thread 28407):
#0 0x00007f2d39546633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3a314f36 in epoll_dispatch (base=0x5ca2a00, tv=<value optimized out>) at epoll.c:404
#2 0x00007f2d3a300394 in event_base_loop (base=0x5ca2a00, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x143d6e8) at daemon/thread.c:301
#4 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 12 (Thread 28408):
#0 0x00007f2d39546633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3a314f36 in epoll_dispatch (base=0x5ca2780, tv=<value optimized out>) at epoll.c:404
#2 0x00007f2d3a300394 in event_base_loop (base=0x5ca2780, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x143d7e0) at daemon/thread.c:301
#4 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 11 (Thread 28404):
#0 0x00007f2d39546633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3a314f36 in epoll_dispatch (base=0x5ca2500, tv=<value optimized out>) at epoll.c:404
#2 0x00007f2d3a300394 in event_base_loop (base=0x5ca2500, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x143d400) at daemon/thread.c:301
#4 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 10 (Thread 28406):
#0 0x00007f2d39546633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3a314f36 in epoll_dispatch (base=0x5ca2c80, tv=<value optimized out>) at epoll.c:404
#2 0x00007f2d3a300394 in event_base_loop (base=0x5ca2c80, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x143d5f0) at daemon/thread.c:301
#4 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 9 (Thread 28402):
#0 0x00007f2d39800f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f2d3833b176 in logger_thead_main (arg=<value optimized out>) at extensions/loggers/file_logger.c:368
#2 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
--Type <return> to continue, or q <return> to quit--
#3 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()

Thread 8 (Thread 28405):
#0 0x00007f2d39546633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3a314f36 in epoll_dispatch (base=0x5ca2280, tv=<value optimized out>) at epoll.c:404
#2 0x00007f2d3a300394 in event_base_loop (base=0x5ca2280, flags=<value optimized out>) at event.c:1558
#3 0x0000000000414b94 in worker_libevent (arg=0x143d4f8) at daemon/thread.c:301
#4 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 7 (Thread 28412):
#0 0x00007f2d39800f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f2d34c3cd68 in wait (this=0x14422d0, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x14422d0, d=...) at src/dispatcher.cc:336
#3 0x00007f2d34c3fd1a in Dispatcher::run (this=0x5cef6c0) at src/dispatcher.cc:173
#4 0x00007f2d34c4061b in launch_dispatcher_thread (arg=0x5cef714) at src/dispatcher.cc:28
#5 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 6 (Thread 28414):
#0 0x00007f2d39800f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f2d34c5bd9f in wait (this=0x5cf8000) at src/syncobject.hh:58
#2 wait (this=0x5cf8000) at src/syncobject.hh:74
#3 wait (this=0x5cf8000) at src/tapconnmap.hh:169
#4 EventuallyPersistentEngine::notifyPendingConnections (this=0x5cf8000) at src/ep_engine.cc:3395
#5 0x00007f2d34c5be83 in EvpNotifyPendingConns (arg=0x5cf8000) at src/ep_engine.cc:1122
#6 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 5 (Thread 28401):
#0 0x00007f2d395372ed in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d394d2798 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f2d394d37be in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f2d394c78fa in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f2d394c67ca in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f2d38d3eb19 in fgets (arg=<value optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<value optimized out>) at extensions/daemon/stdin_check.c:37
#7 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 4 (Thread 28393):
#0 0x00007f2d39546633 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f2d3a314f36 in epoll_dispatch (base=0x5ca2000, tv=<value optimized out>) at epoll.c:404
#2 0x00007f2d3a300394 in event_base_loop (base=0x5ca2000, flags=<value optimized out>) at event.c:1558
--Type <return> to continue, or q <return> to quit--
#3 0x000000000040c2d1 in main (argc=<value optimized out>, argv=<value optimized out>) at daemon/memcached.c:7912

Thread 3 (Thread 28413):
#0 0x00007f2d39800f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f2d34c3cd68 in wait (this=0x1442240, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x1442240, d=...) at src/dispatcher.cc:336
#3 0x00007f2d34c3fd1a in Dispatcher::run (this=0x5cef500) at src/dispatcher.cc:173
#4 0x00007f2d34c4061b in launch_dispatcher_thread (arg=0x5cef554) at src/dispatcher.cc:28
#5 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 2 (Thread 28411):
#0 0x00007f2d39800f2b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f2d34c3cd68 in wait (this=0x1442360, d=...) at src/syncobject.hh:58
#2 IdleTask::run (this=0x1442360, d=...) at src/dispatcher.cc:336
#3 0x00007f2d34c3fd1a in Dispatcher::run (this=0x5cef880) at src/dispatcher.cc:173
#4 0x00007f2d34c4061b in launch_dispatcher_thread (arg=0x5cef8d4) at src/dispatcher.cc:28
#5 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 1 (Thread 28410):
#0 0x00007f2d34a116db in flush_buffer (handle=<value optimized out>, buf=0x7f2d33df2490, nbyte=8, offset=180315) at src/iobuffer.c:106
#1 buffered_pwrite (handle=<value optimized out>, buf=0x7f2d33df2490, nbyte=8, offset=180315) at src/iobuffer.c:331
#2 0x00007f2d34a10434 in raw_write (db=0x5ca8620, buf=0x7f2d33df24e0, pos=<value optimized out>, disk_size=<value optimized out>)
at src/couch_file_write.c:38
#3 db_write_buf (db=0x5ca8620, buf=0x7f2d33df24e0, pos=<value optimized out>, disk_size=<value optimized out>)
at src/couch_file_write.c:98
#4 0x00007f2d34a10629 in db_write_buf_compressed (db=0x5ca8620, buf=0xf528048, pos=0x7f2d33df25c8, disk_size=0x7f2d33df25d8)
at src/couch_file_write.c:136
#5 0x00007f2d34a0f9eb in write_doc (db=<value optimized out>, docs=0x1436420, infos=0x1436288, numdocs=<value optimized out>, options=1)
at src/couch_save.c:45
#6 add_doc_to_update_list (db=<value optimized out>, docs=0x1436420, infos=0x1436288, numdocs=<value optimized out>, options=1)
at src/couch_save.c:272
#7 couchstore_save_documents (db=<value optimized out>, docs=0x1436420, infos=0x1436288, numdocs=<value optimized out>, options=1)
at src/couch_save.c:344
#8 0x00007f2d34cba107 in CouchKVStore::saveDocs (this=0x5d10000, vbid=212, rev=<value optimized out>, docs=<value optimized out>,
docinfos=0x1436288, docCount=1) at src/couch-kvstore/couch-kvstore.cc:1539
#9 0x00007f2d34cba871 in CouchKVStore::commit2couchstore (this=0x5d10000) at src/couch-kvstore/couch-kvstore.cc:1474
#10 0x00007f2d34cbaba9 in queueItem (this=0x5d10000, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:1606
#11 CouchKVStore::set (this=0x5d10000, itm=..., cb=<value optimized out>) at src/couch-kvstore/couch-kvstore.cc:343

#12 0x00007f2d34c4bdd6 in EventuallyPersistentStore::flushOneDelOrSet (this=0x143f200, qi=..., rejectQueue=<value optimized out>)
at src/ep.cc:2165
#13 0x00007f2d34c50ac4 in EventuallyPersistentStore::flushOne (this=0x143f200, q=0x143f4c0, rejectQueue=0xb6f7a40) at src/ep.cc:2205
#14 0x00007f2d34c50c38 in EventuallyPersistentStore::flushSome (this=0x143f200, q=0x143f4c0, rejectQueue=0xb6f7a40) at src/ep.cc:1854
#15 0x00007f2d34c769d9 in Flusher::doFlush (this=0x14425a0) at src/flusher.cc:245
#16 0x00007f2d34c77455 in Flusher::step (this=0xe2075f0, d=..., tid=...) at src/flusher.cc:158
#17 0x00007f2d34c3fd1a in Dispatcher::run (this=0x5ceea80) at src/dispatcher.cc:173
--Type <return> to continue, or q <return> to quit--
#18 0x00007f2d34c4061b in launch_dispatcher_thread (arg=0xe2075f0) at src/dispatcher.cc:28
#19 0x00007f2d397fbd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007f2d39545fdd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()

[ns_server:info,2012-10-03T7:09:28.066,ns_1@10.3.121.94:<0.32168.25>:mc_connection:run_loop:214]mccouch connection was normally closed
[ns_server:debug,2012-10-03T7:09:28.067,ns_1@10.3.121.94:'ns_memcached-bucket-0':ns_memcached:handle_info:624]Got {'EXIT',<0.32170.25>,{badmatch,

{error,closed}}}. Exiting.
[user:info,2012-10-03T7:09:28.074,ns_1@10.3.121.94:'ns_memcached-bucket-0':ns_memcached:terminate:680]Control connection to memcached on 'ns_1@10.3.121.94' disconnected: {badmatch,
{error,
closed}}
[error_logger:error,2012-10-03T7:09:28.074,ns_1@10.3.121.94:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: erlang:apply/2
pid: <0.32170.25>
registered_name: []
exception error: no match of right hand side value {error,closed}

in function mc_binary:quick_stats_recv/3
in call from mc_binary:quick_stats_loop/5
in call from mc_binary:quick_stats/5
in call from ns_memcached:do_handle_call/3
in call from ns_memcached:worker_loop/3
ancestors: ['ns_memcached-bucket-0','single_bucket_sup-bucket-0',
<0.32153.25>]
messages: []
links: [<0.32167.25>]
dictionary: []
trap_exit: false
status: running
heap_size: 28657
stack_size: 24
reductions: 2877355
neighbours:

[user:info,2012-10-03T7:09:28.074,ns_1@10.3.121.94:ns_port_memcached:ns_port_server:handle_info:107]Port server memcached on node 'ns_1@10.3.121.94' exited with status 139. Restarting. Messages: Wed Oct 3 07:09:07.390733 PDT 3: TAP (Consumer) eq_tapq:anon_943 - disconnected
Wed Oct 3 07:09:07.498627 PDT 3: Schedule cleanup of "eq_tapq:anon_942"
Wed Oct 3 07:09:07.498687 PDT 3: Schedule cleanup of "eq_tapq:anon_943"
Wed Oct 3 07:09:07.498682 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:07.804338 PDT 3: TAP (Consumer) eq_tapq:anon_944 - disconnected
Wed Oct 3 07:09:07.925300 PDT 3: TAP (Consumer) eq_tapq:anon_945 - disconnected
Wed Oct 3 07:09:08.020906 PDT 3: Schedule cleanup of "eq_tapq:anon_944"
Wed Oct 3 07:09:08.020986 PDT 3: Schedule cleanup of "eq_tapq:anon_945"
Wed Oct 3 07:09:08.020980 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:08.404868 PDT 3: TAP (Consumer) eq_tapq:anon_946 - disconnected
Wed Oct 3 07:09:08.561115 PDT 3: TAP (Consumer) eq_tapq:anon_947 - disconnected
Wed Oct 3 07:09:08.655894 PDT 3: Schedule cleanup of "eq_tapq:anon_946"
Wed Oct 3 07:09:08.655922 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:08.655968 PDT 3: Schedule cleanup of "eq_tapq:anon_947"
Wed Oct 3 07:09:09.018520 PDT 3: TAP (Consumer) eq_tapq:anon_948 - disconnected
Wed Oct 3 07:09:09.180208 PDT 3: TAP (Consumer) eq_tapq:anon_949 - disconnected
Wed Oct 3 07:09:09.310722 PDT 3: Schedule cleanup of "eq_tapq:anon_948"
Wed Oct 3 07:09:09.310749 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:09.310803 PDT 3: Schedule cleanup of "eq_tapq:anon_949"
Wed Oct 3 07:09:10.667048 PDT 3: TAP (Consumer) eq_tapq:anon_950 - disconnected
Wed Oct 3 07:09:10.802652 PDT 3: TAP (Consumer) eq_tapq:anon_951 - disconnected
Wed Oct 3 07:09:10.893904 PDT 3: Schedule cleanup of "eq_tapq:anon_950"
Wed Oct 3 07:09:10.893977 PDT 3: Schedule cleanup of "eq_tapq:anon_951"
Wed Oct 3 07:09:10.893940 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:12.259851 PDT 3: TAP (Consumer) eq_tapq:anon_952 - disconnected
Wed Oct 3 07:09:12.376843 PDT 3: TAP (Consumer) eq_tapq:anon_953 - disconnected
Wed Oct 3 07:09:12.485074 PDT 3: Schedule cleanup of "eq_tapq:anon_952"
Wed Oct 3 07:09:12.485107 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:12.485148 PDT 3: Schedule cleanup of "eq_tapq:anon_953"
Wed Oct 3 07:09:12.849159 PDT 3: TAP (Consumer) eq_tapq:anon_954 - disconnected
Wed Oct 3 07:09:13.002880 PDT 3: TAP (Consumer) eq_tapq:anon_955 - disconnected
Wed Oct 3 07:09:13.109536 PDT 3: Schedule cleanup of "eq_tapq:anon_954"
Wed Oct 3 07:09:13.109576 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:13.109626 PDT 3: Schedule cleanup of "eq_tapq:anon_955"
Wed Oct 3 07:09:13.529732 PDT 3: TAP (Consumer) eq_tapq:anon_956 - disconnected
Wed Oct 3 07:09:13.714518 PDT 3: TAP (Consumer) eq_tapq:anon_957 - disconnected
Wed Oct 3 07:09:13.815423 PDT 3: Schedule cleanup of "eq_tapq:anon_956"
Wed Oct 3 07:09:13.815489 PDT 3: Schedule cleanup of "eq_tapq:anon_957"
Wed Oct 3 07:09:13.815685 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:14.200683 PDT 3: TAP (Consumer) eq_tapq:anon_958 - disconnected
Wed Oct 3 07:09:14.336430 PDT 3: TAP (Consumer) eq_tapq:anon_959 - disconnected
Wed Oct 3 07:09:14.425384 PDT 3: Schedule cleanup of "eq_tapq:anon_958"
Wed Oct 3 07:09:14.425456 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:14.425466 PDT 3: Schedule cleanup of "eq_tapq:anon_959"
Wed Oct 3 07:09:15.601665 PDT 3: TAP (Consumer) eq_tapq:anon_960 - disconnected
Wed Oct 3 07:09:15.720572 PDT 3: TAP (Consumer) eq_tapq:anon_961 - disconnected
Wed Oct 3 07:09:15.816248 PDT 3: Schedule cleanup of "eq_tapq:anon_960"
Wed Oct 3 07:09:15.816297 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:15.816359 PDT 3: Schedule cleanup of "eq_tapq:anon_961"
Wed Oct 3 07:09:16.225778 PDT 3: TAP (Consumer) eq_tapq:anon_962 - disconnected
Wed Oct 3 07:09:16.362276 PDT 3: TAP (Consumer) eq_tapq:anon_963 - disconnected
Wed Oct 3 07:09:16.448418 PDT 3: Schedule cleanup of "eq_tapq:anon_962"
Wed Oct 3 07:09:16.448512 PDT 3: Schedule cleanup of "eq_tapq:anon_963"
Wed Oct 3 07:09:16.448533 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:16.752436 PDT 3: TAP (Consumer) eq_tapq:anon_964 - disconnected
Wed Oct 3 07:09:16.908300 PDT 3: TAP (Consumer) eq_tapq:anon_965 - disconnected
Wed Oct 3 07:09:17.076167 PDT 3: Schedule cleanup of "eq_tapq:anon_964"
Wed Oct 3 07:09:17.076241 PDT 3: Schedule cleanup of "eq_tapq:anon_965"
Wed Oct 3 07:09:17.076465 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:17.422470 PDT 3: TAP (Consumer) eq_tapq:anon_966 - disconnected
Wed Oct 3 07:09:17.558228 PDT 3: TAP (Consumer) eq_tapq:anon_967 - disconnected
Wed Oct 3 07:09:17.660250 PDT 3: Schedule cleanup of "eq_tapq:anon_966"
Wed Oct 3 07:09:17.660290 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:17.660306 PDT 3: Schedule cleanup of "eq_tapq:anon_967"
Wed Oct 3 07:09:19.141698 PDT 3: TAP (Consumer) eq_tapq:anon_968 - disconnected
Wed Oct 3 07:09:19.264492 PDT 3: TAP (Consumer) eq_tapq:anon_969 - disconnected
Wed Oct 3 07:09:19.364848 PDT 3: Schedule cleanup of "eq_tapq:anon_968"
Wed Oct 3 07:09:19.364931 PDT 3: Schedule cleanup of "eq_tapq:anon_969"
Wed Oct 3 07:09:19.365354 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:19.733874 PDT 3: TAP (Consumer) eq_tapq:anon_970 - disconnected
Wed Oct 3 07:09:19.842574 PDT 3: TAP (Consumer) eq_tapq:anon_971 - disconnected
Wed Oct 3 07:09:19.941458 PDT 3: Schedule cleanup of "eq_tapq:anon_970"
Wed Oct 3 07:09:19.941843 PDT 3: Schedule cleanup of "eq_tapq:anon_971"
Wed Oct 3 07:09:19.941791 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:21.354628 PDT 3: TAP (Consumer) eq_tapq:anon_972 - disconnected
Wed Oct 3 07:09:21.480109 PDT 3: TAP (Consumer) eq_tapq:anon_973 - disconnected
Wed Oct 3 07:09:21.576311 PDT 3: Schedule cleanup of "eq_tapq:anon_972"
Wed Oct 3 07:09:21.576402 PDT 3: Schedule cleanup of "eq_tapq:anon_973"
Wed Oct 3 07:09:21.576465 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:22.841716 PDT 3: TAP (Consumer) eq_tapq:anon_974 - disconnected
Wed Oct 3 07:09:22.952241 PDT 3: TAP (Consumer) eq_tapq:anon_975 - disconnected
Wed Oct 3 07:09:23.040710 PDT 3: Schedule cleanup of "eq_tapq:anon_974"
Wed Oct 3 07:09:23.040738 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:23.040880 PDT 3: Schedule cleanup of "eq_tapq:anon_975"
Wed Oct 3 07:09:23.308185 PDT 3: TAP (Consumer) eq_tapq:anon_976 - disconnected
Wed Oct 3 07:09:23.481399 PDT 3: TAP (Consumer) eq_tapq:anon_977 - disconnected
Wed Oct 3 07:09:23.566951 PDT 3: Schedule cleanup of "eq_tapq:anon_976"
Wed Oct 3 07:09:23.567046 PDT 3: Schedule cleanup of "eq_tapq:anon_977"
Wed Oct 3 07:09:23.567184 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:24.918727 PDT 3: TAP (Consumer) eq_tapq:anon_978 - disconnected
Wed Oct 3 07:09:25.054151 PDT 3: TAP (Consumer) eq_tapq:anon_979 - disconnected
Wed Oct 3 07:09:25.142569 PDT 3: Schedule cleanup of "eq_tapq:anon_978"
Wed Oct 3 07:09:25.142638 PDT 3: Schedule cleanup of "eq_tapq:anon_979"
Wed Oct 3 07:09:25.142835 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:25.503745 PDT 3: TAP (Consumer) eq_tapq:anon_980 - disconnected
Wed Oct 3 07:09:25.631364 PDT 3: TAP (Consumer) eq_tapq:anon_981 - disconnected
Wed Oct 3 07:09:25.724000 PDT 3: Schedule cleanup of "eq_tapq:anon_980"
Wed Oct 3 07:09:25.724036 PDT 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.92 - Sending TAP_OPAQUE with command "complete_vb_filter_change" and vbucket 0
Wed Oct 3 07:09:25.724083 PDT 3: Schedule cleanup of "eq_tapq:anon_981"
Wed Oct 3 07:09:27.022598 PDT 3: TAP (Consumer) eq_tapq:anon_982 - disconnected
[stats:error,2012-10-03T7:09:28.077,ns_1@10.3.121.94:<0.32178.25>:stats_collector:handle_info:106]Exception in stats collector: {exit,
badmatch,{error,closed,
{gen_server,call,
['ns_memcached-bucket-0',

{stats,<<>>}

,
60000]}},
[

{gen_server,call,3}

,

{ns_memcached,do_call,3}

,

{stats_collector,grab_all_stats,1}

,

{stats_collector,handle_info,2}

,

{gen_server,handle_msg,5}

,

{proc_lib,init_p_do_apply,3}

]}

[error_logger:error,2012-10-03T7:09:28.079,ns_1@10.3.121.94:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: erlang:apply/2
pid: <0.32169.25>
registered_name: []
exception error: no match of right hand side value

{error,closed}

in function mc_binary:quick_stats_recv/3
in call from mc_binary:quick_stats_loop/5
in call from mc_binary:quick_stats/5
in call from ns_memcached:do_handle_call/3
in call from ns_memcached:worker_loop/3
ancestors: ['ns_memcached-bucket-0','single_bucket_sup-bucket-0',
<0.32153.25>]
messages: []
links: [<0.32167.25>]
dictionary: []
trap_exit: false
status: running
heap_size: 196418
stack_size: 24
reductions: 1837035
neighbours:

[ns_server:info,2012-10-03T7:09:28.081,ns_1@10.3.121.94:<0.32103.25>:supervisor_cushion:handle_info:62]Cushion managed supervisor for memcached failed:

{abnormal,139}

Control connection to memcached on 'ns_1@10.3.121.94' disconnected: {badmatch,

{error, closed}

}