Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-353

Add node IP to error messages so that users can isolate issues easier

    XMLWordPrintable

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.3.1
    • library

    Description

      From the user:

      Today we experienced an issue with one of our Couchbase nodes in production that caused a major response speed decrease. The root cause was a human error on our side, but the issue showed us another problem in connection pool refactoring in latest version of .NET SDK.

      This is error we get from .NET SDK 1.2.9:
      <GetOperationResult Success="False" StatusCode="145" CAS="0">
      <Message>Pool is full, timeouting. 10.205.10.165:11210</Message>
      <Exception><![CDATA[System.TimeoutException: The operation has timed out.]]></Exception>
      <InnerResult>
      <PooledSocketResult Success="False" StatusCode="145">
      <Message>Pool is full, timeouting. 10.205.10.165:11210</Message>
      <Exception><![CDATA[System.TimeoutException: The operation has timed out.]]></Exception>
      </PooledSocketResult>
      </InnerResult>
      </GetOperationResult>

      This is equivalent error that we got from .NET SDK 1.3.0:
      <GetOperationResult Success="False" StatusCode="145" CAS="0">
      <Message>Queue Timeout.</Message>
      <Exception><![CDATA[Couchbase.Exceptions.QueueTimeoutException: Timeout occured while waiting for a socket.
      Your current configuration for queueTmeout is 00:00:02.5000000
      Your current configuration for maxPoolSize is 6
      Try increasing queueTimeout or increasing using maxPoolSize in your configuration.
      ---> System.InvalidOperationException: Queue empty.
      at System.Collections.Generic.Queue`1.Dequeue()
      at Couchbase.SocketPool.Acquire()
      — End of inner exception stack trace —
      at Couchbase.SocketPool.Acquire()
      at Couchbase.CouchbaseNode.Execute(IOperation op)]]></Exception>
      <InnerResult>
      <BinaryOperationResult Success="False" StatusCode="145" CAS="0">
      <Message>Queue Timeout.</Message>
      <Exception><![CDATA[Couchbase.Exceptions.QueueTimeoutException: Timeout occured while waiting for a socket.
      Your current configuration for queueTmeout is 00:00:02.5000000
      Your current configuration for maxPoolSize is 6
      Try increasing queueTimeout or increasing using maxPoolSize in your configuration.
      ---> System.InvalidOperationException: Queue empty.
      at System.Collections.Generic.Queue`1.Dequeue()
      at Couchbase.SocketPool.Acquire()
      — End of inner exception stack trace —
      at Couchbase.SocketPool.Acquire()
      at Couchbase.CouchbaseNode.Execute(IOperation op)]]></Exception>
      </BinaryOperationResult>
      </InnerResult>
      </GetOperationResult>

      While the version 1.3.0 gives us some additional information about our current connection pool configuration, we lost the information about Couchbase node IP which was pivotal in resolution of the issue we had. If all our applications were already running client 1.3.0, we would not find out that the issue is affecting only single node in the production pool.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jmorris Jeff Morris
            jmorris Jeff Morris
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty