Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
2.10.0
-
Security Level: Public
-
Red Hat Enterprise Linux Server release 6.3 (Santiago).
Java 7
Description
I'm studying the following scenario: I have cluster(2 nodes). While get/set operations I do failover on one of nodes. After I do rebalance and I see in logs follow message: "Replacing current streaming node list http://node1:8091/pools, http://node2:8091/pools with http://node1:8091/pools" from com.couchbase.client.CouchbaseConnectionFactoryBuilder$1 (if it helps you, this message from class CouchbaseConnectionFactory, method "updateStoredBaseList").
But after this a see follow messages in logs: "(Memcached IO over
) 2013-11-05 12:20:17.848 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 20056 Key: 2369". I get a lot of similar message but among them sometimes like this "12:21:47,864 ERROR [stderr] (Memcached IO over
{MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}) 2013-11-05 12:21:47.864 INFO com.couchbase.client.CouchbaseConnection: Reconnecting
{QA sa=couchbase-2/node-2:11210, #Rops=0, #Wops=240, #iq=0, topRop=null, topWop=Cmd: 2 Opaque: 21709 Key: 32417 Cas: 0 Exp: 0 Flags: 0 Data Length: 9, toWrite=0, interested=0}12:21:47,962 ERROR [stderr] (Memcached IO over
{MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}) 2013-11-05 12:21:47.962 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening
{QA sa=couchbase-2/node-2:11210, #Rops=0, #Wops=240, #iq=2, topRop=null, topWop=Cmd: 2 Opaque: 21709 Key: 32417 Cas: 0 Exp: 0 Flags: 0 Data Length: 9, toWrite=0, interested=8}, attempt 6."
From this I conclude that the node list on memcached side not been replaced.
From several minutes I see finally messages from log every two seconds:"12:25:00,841 ERROR [stderr] (Memcached IO over
) 2013-11-05 12:25:00.841 INFO com.couchbase.client.CouchbaseConnection: Reconnecting
{QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=0}12:25:00,842 ERROR [stderr] (Memcached IO over
{MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}) 2013-11-05 12:25:00.842 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@13293535
12:25:00,842 ERROR [stderr] (Memcached IO over
) 2013-11-05 12:25:00.842 WARN net.spy.memcached.auth.AuthThreadMonitor: Incomplete authentication interrupted for node
{QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=8}12:25:00,843 ERROR [stderr] (Thread-247) 2013-11-05 12:25:00.843 WARN net.spy.memcached.auth.AuthThread: Authentication failed to couchbase-2/node-2:11210
12:25:00,949 ERROR [stderr] (Memcached IO over
) 2013-11-05 12:25:00.948 ERROR net.spy.memcached.protocol.binary.SASLStepOperationImpl: Error: Auth failure
12:25:00,949 ERROR [stderr] (Memcached IO over
) 2013-11-05 12:25:00.949 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: SASL steps operation
12:25:00,950 ERROR [stderr] (Memcached IO over
) 2013-11-05 12:25:00.949 INFO com.couchbase.client.CouchbaseConnection: Reconnection due to exception handling a memcached operation on
{QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=1}. This may be due to an authentication failure.
12:25:00,950 ERROR [stderr] (Memcached IO over
) OperationException: SERVER: Auth failure
12:25:00,950 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:166)
12:25:00,950 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:176)
12:25:00,951 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.finishedPayload(SASLBaseOperationImpl.java:95)
12:25:00,951 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:162)
12:25:00,951 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.readFromBuffer(SASLBaseOperationImpl.java:41)
12:25:00,951 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:628)
12:25:00,952 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:543)
12:25:00,952 ERROR [stderr] (Memcached IO over
) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:322)
12:25:00,952 ERROR [stderr] (Memcached IO over
) at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:288)
12:25:00,952 ERROR [stderr] (Memcached IO over
) 2013-11-05 12:25:00.952 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening
{QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=1}, attempt 0.
12:25:01,049 ERROR [stderr] (Thread-248) 2013-11-05 12:25:01.049 WARN net.spy.memcached.auth.AuthThread: Authentication failed to couchbase-2/node-2:11210". While all this is happening my sets/gets operations don't reach Couchbase.