2014-06-11 12:51:48.671 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:48.869 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for valueException at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:48.870 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:48.870 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:48.870 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:48.871 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:48.872 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. net.spy.memcached.OperationTimeoutException: Timeout waiting for value Exception Exception at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.070 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.070 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.070 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.071 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.071 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.072 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.270 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more Exception 2014-06-11 12:51:49.271 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.271 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.272 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.272 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.273 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.468 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=node1/10.23.40.89:11210, #Rops=0, #Wops=58, #iq=0, topRop=null, topWop=Cmd: 0 Opaque: 111095 Key: 15b552fc-382c-410e-b0d4-d9408770089d, toWrite=0, interested=0} Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.471 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.471 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.472 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.472 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.473 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.473 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.536 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 111153 Key: 15b552fc-382c-410e-b0d4-d9408770089d 2014-06-11 12:51:49.536 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 111154 Key: 15b552fc-382c-410e-b0d4-d9408770089d Exception in thread "Thread-13" java.util.concurrent.CancellationException: Cancelled at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:96) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) 2014-06-11 12:51:49.537 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 111155 Key: 15b552fc-382c-410e-b0d4-d9408770089d 2014-06-11 12:51:49.537 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 111156 Key: 15b552fc-382c-410e-b0d4-d9408770089d 2014-06-11 12:51:49.537 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 111157 Key: 15b552fc-382c-410e-b0d4-d9408770089d 2014-06-11 12:51:49.537 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 111158 Key: 15b552fc-382c-410e-b0d4-d9408770089d Exception in thread "Thread-8" java.util.concurrent.CancellationException: Cancelled at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:177) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:96) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) 2014-06-11 12:51:49.537 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: SASL mechs operation 2014-06-11 12:51:49.537 WARN net.spy.memcached.auth.AuthThread: Received non-success response for SASL mechs: {OperationStatus success=false: cancelled} 2014-06-11 12:51:49.538 INFO com.couchbase.client.CouchbaseConnection: Reconnecting due to exception on {QA sa=node1/10.23.40.89:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=1} java.io.IOException: Disconnected unexpected, will reconnect. at net.spy.memcached.MemcachedConnection.handleReadsWhenChannelEndOfStream(MemcachedConnection.java:910) at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:822) at net.spy.memcached.MemcachedConnection.handleReadsAndWrites(MemcachedConnection.java:719) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:682) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:436) at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:321) 2014-06-11 12:51:49.538 WARN net.spy.memcached.auth.AuthThread: Authentication failed to node1/10.23.40.89:11210, got empty SASL auth mech list. 2014-06-11 12:51:49.538 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening {QA sa=node1/10.23.40.89:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=1}, attempt 0. Exception in thread "Thread-15" java.lang.IllegalStateException: Got empty SASL auth mech list. at net.spy.memcached.auth.AuthThread.run(AuthThread.java:149) Exceptionnet.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.672 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more net.spy.memcached.OperationTimeoutException: Timeout waiting for value Exception Exception at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.673 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:49.673 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. Exception net.spy.memcached.OperationTimeoutException: Timeout waiting for value at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:868) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:850) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:103) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.TimeoutException: No replica get future returned with success before timeout. at com.couchbase.client.internal.ReplicaGetFuture.get(ReplicaGetFuture.java:111) at com.couchbase.client.CouchbaseClient.getFromReplica(CouchbaseClient.java:861) ... 4 more 2014-06-11 12:51:49.674 WARN com.couchbase.client.CouchbaseConnection: Node expected to receive data is inactive. This could be due to a failure within the cluster. Will check for updated configuration. Key without a configured node is: 15b552fc-382c-410e-b0d4-d9408770089d. 2014-06-11 12:51:51.539 INFO com.couchbase.client.CouchbaseConnection: Reconnecting {QA sa=node1/10.23.40.89:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} 2014-06-11 12:51:51.777 INFO net.spy.memcached.auth.AuthThread: Authenticated to node1/10.23.40.89:11210 Exception in thread "Thread-11" java.lang.RuntimeException: Exception waiting for value at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1237) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:96) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.ExecutionException: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:180) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) ... 4 more Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 ... 7 more Exception in thread "Thread-9" java.lang.RuntimeException: Exception waiting for value at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1237) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:96) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.ExecutionException: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:180) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) ... 4 more Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 ... 7 more Exception in thread "Thread-10" java.lang.RuntimeException: Exception waiting for value at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1237) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:96) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.ExecutionException: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:180) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) ... 4 more Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 ... 7 more Exception in thread "Thread-12" java.lang.RuntimeException: Exception waiting for value at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1237) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1257) at com.couchbase.failover.testing.CouchbaseClientTester.couchbaseGet(CouchbaseClientTester.java:96) at com.couchbase.failover.testing.CouchbaseClientTester.access$0(CouchbaseClientTester.java:90) at com.couchbase.failover.testing.CouchbaseClientTester$TestThread.run(CouchbaseClientTester.java:85) Caused by: java.util.concurrent.ExecutionException: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:180) at net.spy.memcached.internal.GetFuture.get(GetFuture.java:69) at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1230) ... 4 more Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Operation timed out. - failing node: node1/10.23.40.89:11210 ... 7 more