Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42607

TLS handshake fails if node certificate requires more than 8K to transmit

    XMLWordPrintable

    Details

    • Triage:
      Triaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown
    • Sprint:
      KV Sprint 2020-Oct, KV-Engine Sprint 2020-Dec, KV-Engine 2021-Jan

      Description

      Summary

      During TLS handshake with the Data Service, if the node certificate requires more than 8192 bytes to transmit then the handshake can fail with the following error:

      WARNING 634: ERROR: SSL_accept returned -1 with error 3
      INFO 634 Closing connection [ 1.2.3.4:55555 - 5.6.7.8:11207 (not authenticated) ] due to read error: Connection reset by peer
      

      Details

      The KV-Engine SSL handshake code fails to handle one of the possible temporary status codes from SSL_accept(), namely SSL_ERROR_WANT_WRITE which occurs when OpenSSL has consumed the BIO send buffer but still has more data it wishes to write. Given the BIO buffer size is 8192 bytes, if sending the node certificate requires more than 8192B then SSL_ERROR_WANT_WRITE is returned by OpenSSL.

      Node certificates which are in excess of 8kB - for example those which contain a large number of Subject Alternative Names (SANs) - can encounter this problem.

      Note: Version 7.0 and upwards is not affected as has a different implementation of the TLS handshake.

      Workaround.

      Reduce the size of the node certificate - for example instead of using a single node certificate (with all the different cluster node hostnames listed as SANs), configure individual per-node certificates with just the specific node's hostname.

      The exact certificate size limit is hard to precisely specify, given the certificate is not sent as-is over the TCP/IP connection, however empirically certificates larger than 6kB can encounter this issue (as they can increase to 8kB in size when transmitted).

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-4255 contains kv_engine commit b4e3057 with commit message:
          MB-42607 [2/2]: Handle SSL_accept returning SSL_ERROR_WANT_WRITE

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4255 contains kv_engine commit b4e3057 with commit message: MB-42607 [2/2] : Handle SSL_accept returning SSL_ERROR_WANT_WRITE
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-4255 contains kv_engine commit 35c21b1 with commit message:
          MB-42607 [1/2]: Make bio_drain_buffer_sz dynamic

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4255 contains kv_engine commit 35c21b1 with commit message: MB-42607 [1/2] : Make bio_drain_buffer_sz dynamic
          Hide
          ritam.sharma Ritam Sharma added a comment -

          Verified for CC as well for 9900 bytes - 7.0.0-4256

          Show
          ritam.sharma Ritam Sharma added a comment - Verified for CC as well for 9900 bytes - 7.0.0-4256
          Hide
          ritam.sharma Ritam Sharma added a comment -

          Daniel Owen - Add CC as fix version for this ticket.

          Show
          ritam.sharma Ritam Sharma added a comment - Daniel Owen - Add CC as fix version for this ticket.
          Hide
          drigby Dave Rigby added a comment -

          Note: master branch (7.0.0) isn't affected by this issue as the changes for out-of-order responses mean we no longer use the same SSL handling functionality, and hence no code change was made to CC.

          However, from the outside world this sublety is arguably irrelevant, so leaving CC as a "fix" version.

          Show
          drigby Dave Rigby added a comment - Note: master branch (7.0.0) isn't affected by this issue as the changes for out-of-order responses mean we no longer use the same SSL handling functionality, and hence no code change was made to CC. However, from the outside world this sublety is arguably irrelevant, so leaving CC as a "fix" version.

            People

            Assignee:
            drigby Dave Rigby
            Reporter:
            drigby Dave Rigby
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                  PagerDuty