Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: .backlog1.x
    • Component/s: Documentation
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Debian GNU/Linux 6.0 (amd64), Couchbase community edition.

      Description

      Let's assume I have:

      • a cluster made of 4 couchbase nodes (couchbase1-couchbase4);
      • 10 instances of client application (client1-client10).

      In my client Java application I connect to the cluster using login, password and the following server list string:

      "http://couchbase1:8091/pools,http://couchbase2:8091/pools,http://couchbase3:8091/pools,http://couchbase4:8091/pools"
      

      When I start four instances of client application on client1-client4 – everything is going OK. But when I am trying to start 5-th instance of client application on client5 – I get the following error:

      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.564 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting {QA sa=couchbase1/192.168.0.129:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=0}
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.565 INFO com.couchbase.client.CouchbaseConnection:  Connection state changed for sun.nio.ch.SelectionKeyImpl@28a3001c
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.565 WARN net.spy.memcached.auth.AuthThreadMonitor:  Incomplete authentication interrupted for node {QA sa=couchbase1/192.168.0.129:11210, #Rops=0, #Wops=1, #iq=0, topRop=null, topWop=SASL steps operation, toWrite=0, interested=8}
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.565 WARN net.spy.memcached.auth.AuthThread:  Authentication failed to couchbase1/192.168.0.129:11210
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.566 INFO com.couchbase.client.CouchbaseConnection:  Reconnecting due to exception on {QA sa=couchbase1/192.168.0.129:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL auth operation, topWop=null, toWrite=0, interested=1}
      Mar 29 12:56:28 client1:    at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:453)
      Mar 29 12:56:28 client1:    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:381)
      Mar 29 12:56:28 client1:    at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:243)
      Mar 29 12:56:28 client1:    at com.couchbase.client.CouchbaseConnection.run(CouchbaseConnection.java:229)
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.566 WARN com.couchbase.client.CouchbaseConnection:  Closing, and reopening {QA sa=couchbase1/192.168.0.129:11210, #Rops=1, #Wops=0, #iq=0, topRop=SASL auth operation, topWop=null, toWrite=0, interested=1}, attempt 0.
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.566 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl:  Discarding partially completed op: SASL auth operation
      Mar 29 12:56:28 client1: 2013-03-29 12:56:28.666 WARN net.spy.memcached.auth.AuthThread:  Authentication failed to couchbase1/192.168.0.129:11210
      

      So, if I shutdown one of client1-client4 – this moment client5 connects to the cluster without problems.

      PS: No matter if I make cluster from 2 nodes. The behaveour is the same – 5-th node get the same error.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        andrew.kulikov Andrew Kulikov added a comment -

        We have solved our problem! The reason was in limit for number of opened files. Have you ever tested couchbase with pam_limits enabled? If so, default configuration (nofile=1024) is suitable only for 3 clients simultaneously connected. 4-th client fails. We have greatly increased nofile parameter for couchbase user in limits.conf – and the problem was solved.

        I think you should add this issue in your documentation. Also, you should write an error in logfile – if this limit (for open files or any other) is reached.

        Show
        andrew.kulikov Andrew Kulikov added a comment - We have solved our problem! The reason was in limit for number of opened files. Have you ever tested couchbase with pam_limits enabled? If so, default configuration (nofile=1024) is suitable only for 3 clients simultaneously connected. 4-th client fails. We have greatly increased nofile parameter for couchbase user in limits.conf – and the problem was solved. I think you should add this issue in your documentation. Also, you should write an error in logfile – if this limit (for open files or any other) is reached.
        Hide
        daschl Michael Nitschinger added a comment -

        Thanks for reporting your findings! I'll change it to a docs enhancement!

        Show
        daschl Michael Nitschinger added a comment - Thanks for reporting your findings! I'll change it to a docs enhancement!

          People

          • Assignee:
            daschl Michael Nitschinger
            Reporter:
            andrew.kulikov Andrew Kulikov
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Gerrit Reviews

              There are no open Gerrit changes