$ java -ea -jar ConcurrentCouchbaseCheck.jar 1200 10.3.2.28
May 30, 2012 4:08:30 PM net.spy.memcached.MemcachedConnection createConnections
INFO: Added {QA sa=/10.3.2.28:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
May 30, 2012 4:08:30 PM net.spy.memcached.MemcachedConnection createConnections
INFO: Added {QA sa=/10.3.2.32:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
May 30, 2012 4:08:30 PM net.spy.memcached.MemcachedConnection createConnections
INFO: Added {QA sa=/10.3.2.33:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
May 30, 2012 4:08:31 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@54cbf30e
May 30, 2012 4:08:31 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@19c247a0
May 30, 2012 4:08:31 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@21ec6696
May 30, 2012 4:08:31 PM com.couchbase.demo.CouchbaseCheck testThem
INFO: Took 529.435ms to connect.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-1 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-2 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-3 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-4 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-5 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-6 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-8 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-9 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-10 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-7 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-13 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-11 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-14 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-15 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-16 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-17 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-18 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-19 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-20 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-12 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-21 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-22 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-23 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-24 starting.
May 30, 2012 4:08:31 PM com.couchbase.demo.SleepingTester run
INFO: Thread pool-3-thread-25 starting.
May 30, 2012 4:10:03 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Reconnecting due to exception on {QA sa=10.3.2.33/10.3.2.33:11210, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 177046 Key: pool-3-thread-23-i3031 Cas: 0 Exp: 0 Flags: 0 Data Length: 22, topWop=null, toWrite=0, interested=1}
java.io.IOException: Disconnected unexpected, will reconnect.
	at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:452)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:380)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
	at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:229)
May 30, 2012 4:10:03 PM net.spy.memcached.MemcachedConnection queueReconnect
WARNING: Closing, and reopening {QA sa=10.3.2.33/10.3.2.33:11210, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 177046 Key: pool-3-thread-23-i3031 Cas: 0 Exp: 0 Flags: 0 Data Length: 22, topWop=null, toWrite=0, interested=1}, attempt 0.
May 30, 2012 4:10:03 PM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend
WARNING: Discarding partially completed op: Cmd: 1 Opaque: 177046 Key: pool-3-thread-23-i3031 Cas: 0 Exp: 0 Flags: 0 Data Length: 22
May 30, 2012 4:10:03 PM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend
WARNING: Discarding partially completed op: Cmd: 1 Opaque: 177047 Key: pool-3-thread-17-i3023 Cas: 0 Exp: 0 Flags: 0 Data Length: 22
May 30, 2012 4:10:03 PM com.couchbase.client.CouchbaseConnection addOperation
WARNING: Node exepcted to receive data is inactive.  This could be due to a failure within the cluster.  Will check for updated configuration.  Key without a configured node is: pool-3-thread-18-i3032.
May 30, 2012 4:10:03 PM com.couchbase.client.CouchbaseConnectionFactory checkConfigUpdate
WARNING: No reconnect required, though check requested. Current config check is 1 out of a threshold of 10.
Execution exception java.lang.RuntimeException: Cancelled
Execution exception java.lang.RuntimeException: Cancelled
May 30, 2012 4:10:03 PM com.couchbase.client.CouchbaseConnection addOperation
WARNING: Node exepcted to receive data is inactive.  This could be due to a failure within the cluster.  Will check for updated configuration.  Key without a configured node is: pool-3-thread-19-i3023.
May 30, 2012 4:10:03 PM com.couchbase.client.CouchbaseConnectionFactory checkConfigUpdate
WARNING: No reconnect required, though check requested. Current config check is 2 out of a threshold of 10.
May 30, 2012 4:10:05 PM net.spy.memcached.MemcachedConnection attemptReconnects
INFO: Reconnecting {QA sa=10.3.2.33/10.3.2.33:11210, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 177053 Key: pool-3-thread-18-i3032 Cas: 0 Exp: 0 Flags: 0 Data Length: 22, toWrite=0, interested=0}
May 30, 2012 4:10:05 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@239a029e
May 30, 2012 4:10:05 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Reconnecting due to failure to connect to {QA sa=10.3.2.33/10.3.2.33:11210, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 177053 Key: pool-3-thread-18-i3032 Cas: 0 Exp: 0 Flags: 0 Data Length: 22, toWrite=0, interested=0}
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
	at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
	at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:229)
May 30, 2012 4:10:05 PM net.spy.memcached.MemcachedConnection queueReconnect
WARNING: Closing, and reopening {QA sa=10.3.2.33/10.3.2.33:11210, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 177053 Key: pool-3-thread-18-i3032 Cas: 0 Exp: 0 Flags: 0 Data Length: 22, toWrite=0, interested=0}, attempt 1.
May 30, 2012 4:10:09 PM net.spy.memcached.MemcachedConnection attemptReconnects
INFO: Reconnecting {QA sa=10.3.2.33/10.3.2.33:11210, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 177053 Key: pool-3-thread-18-i3032 Cas: 0 Exp: 0 Flags: 0 Data Length: 22, toWrite=0, interested=0}
May 30, 2012 4:10:09 PM net.spy.memcached.MemcachedConnection handleIO
INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@68de462
Runtime exception net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 10.3.2.33/10.3.2.33:11210
Runtime exception net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: 10.3.2.33/10.3.2.33:11210