Uploaded image for project: 'Spymemcached Java Client'
  1. Spymemcached Java Client
  2. SPY-143

Possible issue with operations after failover and rebalance.

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.10.0
    • Fix Version/s: None
    • Component/s: library
    • Security Level: Public
    • Environment:
      Red Hat Enterprise Linux Server release 6.3 (Santiago).
      Java 7

      Description

      I'm studying the following scenario: I have cluster(2 nodes). While get/set operations I do failover on one of nodes. After I do rebalance and I see in logs follow message: "Replacing current streaming node list http://node1:8091/pools, http://node2:8091/pools with http://node1:8091/pools" from com.couchbase.client.CouchbaseConnectionFactoryBuilder$1 (if it helps you, this message from class CouchbaseConnectionFactory, method "updateStoredBaseList").
      But after this a see follow messages in logs: "(Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:20:17.848 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: Cmd: 0 Opaque: 20056 Key: 2369". I get a lot of similar message but among them sometimes like this "12:21:47,864 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:21:47.864 INFO com.couchbase.client.CouchbaseConnection: Reconnecting

      {QA sa=couchbase-2/node-2:11210, #Rops=0, #Wops=240, #iq=0, topRop=null, topWop=Cmd: 2 Opaque: 21709 Key: 32417 Cas: 0 Exp: 0 Flags: 0 Data Length: 9, toWrite=0, interested=0}

      12:21:47,962 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:21:47.962 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening

      {QA sa=couchbase-2/node-2:11210, #Rops=0, #Wops=240, #iq=2, topRop=null, topWop=Cmd: 2 Opaque: 21709 Key: 32417 Cas: 0 Exp: 0 Flags: 0 Data Length: 9, toWrite=0, interested=8}

      , attempt 6."

      From this I conclude that the node list on memcached side not been replaced.
      From several minutes I see finally messages from log every two seconds:"12:25:00,841 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.841 INFO com.couchbase.client.CouchbaseConnection: Reconnecting

      {QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=0}

      12:25:00,842 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.842 INFO com.couchbase.client.CouchbaseConnection: Connection state changed for sun.nio.ch.SelectionKeyImpl@13293535
      12:25:00,842 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.842 WARN net.spy.memcached.auth.AuthThreadMonitor: Incomplete authentication interrupted for node

      {QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=8}

      12:25:00,843 ERROR [stderr] (Thread-247) 2013-11-05 12:25:00.843 WARN net.spy.memcached.auth.AuthThread: Authentication failed to couchbase-2/node-2:11210
      12:25:00,949 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.948 ERROR net.spy.memcached.protocol.binary.SASLStepOperationImpl: Error: Auth failure
      12:25:00,949 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.949 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Discarding partially completed op: SASL steps operation
      12:25:00,950 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.949 INFO com.couchbase.client.CouchbaseConnection: Reconnection due to exception handling a memcached operation on

      {QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=1}

      . This may be due to an authentication failure.
      12:25:00,950 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) OperationException: SERVER: Auth failure
      12:25:00,950 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.protocol.BaseOperationImpl.handleError(BaseOperationImpl.java:166)
      12:25:00,950 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.protocol.binary.OperationImpl.finishedPayload(OperationImpl.java:176)
      12:25:00,951 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.finishedPayload(SASLBaseOperationImpl.java:95)
      12:25:00,951 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.protocol.binary.OperationImpl.readFromBuffer(OperationImpl.java:162)
      12:25:00,951 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.protocol.binary.SASLBaseOperationImpl.readFromBuffer(SASLBaseOperationImpl.java:41)
      12:25:00,951 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:628)
      12:25:00,952 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:543)
      12:25:00,952 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:322)
      12:25:00,952 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:288)
      12:25:00,952 ERROR [stderr] (Memcached IO over

      {MemcachedConnection to couchbase-1/node-1:11210 couchbase-2/node-2:11210}

      ) 2013-11-05 12:25:00.952 WARN com.couchbase.client.CouchbaseConnection: Closing, and reopening

      {QA sa= couchbase-2/node-2:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=1}

      , attempt 0.
      12:25:01,049 ERROR [stderr] (Thread-248) 2013-11-05 12:25:01.049 WARN net.spy.memcached.auth.AuthThread: Authentication failed to couchbase-2/node-2:11210". While all this is happening my sets/gets operations don't reach Couchbase.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            daschl Michael Nitschinger
            Reporter:
            vakoroteev Vladislav Koroteev
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Gerrit Reviews

              There are no open Gerrit changes