Uploaded image for project: 'Java Couchbase JVM Core'
  1. Java Couchbase JVM Core
  2. JVMCBC-543

connection may not recover until OS TCP timeout

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.4.2, 1.5.0, 1.5.8
    • 1.5.9
    • Core
    • None
    • 1

    Description

      In situations where the query node (or any HTTP service) goes offline without sending a TCP FIN, because we no longer pipeline (as of 1.4.2), the keepalive may not detect a stale TCP connection as intended.

      To repro, set up continuous workload with 2 query nodes and pool size fixed. Forcibly take one down and restart it. This will cause the TCP connection to go half open, as no TCP FIN will be sent on "crash".

      Observe that the workload is unbalanced after this.

      Note that this may not be observable without fixed workload as new connecitons would be opened within the pool limit. Also, note that the client doesn't detect it as stale because it won't schedule other workload on the connection. Thus this requires a pretty specific set of circumstances to reproduce.

      Workaround
      Tune TCP timeout down to a shorter time interval or add TCP keep-alive at the OS level.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Made progress with this, in a local poc it works -need to cover a couple more edge cases and add testing before this is ready.

          core-io: http://review.couchbase.org/c/95043/
          java-client: http://review.couchbase.org/c/95044/

          daschl Michael Nitschinger added a comment - Made progress with this, in a local poc it works -need to cover a couple more edge cases and add testing before this is ready. core-io: http://review.couchbase.org/c/95043/ java-client: http://review.couchbase.org/c/95044/

          People

            daschl Michael Nitschinger
            ingenthr Matt Ingenthron
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty