TLS handshake fails if node certificate requires more than 8K to transmit

Summary

During TLS handshake with the Data Service, if the node certificate requires more than 8192 bytes to transmit then the handshake can fail with the following error:

WARNING 634: ERROR: SSL_accept returned -1 with error 3

INFO 634 Closing connection [ 1.2.3.4:55555 - 5.6.7.8:11207 (not authenticated) ] due to read error: Connection reset by peer

Details

The KV-Engine SSL handshake code fails to handle one of the possible temporary status codes from SSL_accept(), namely SSL_ERROR_WANT_WRITE which occurs when OpenSSL has consumed the BIO send buffer but still has more data it wishes to write. Given the BIO buffer size is 8192 bytes, if sending the node certificate requires more than 8192B then SSL_ERROR_WANT_WRITE is returned by OpenSSL.

Node certificates which are in excess of 8kB - for example those which contain a large number of Subject Alternative Names (SANs) - can encounter this problem.

Note: Version 7.0 and upwards is not affected as has a different implementation of the TLS handshake.

Workaround.

Reduce the size of the node certificate - for example instead of using a single node certificate (with all the different cluster node hostnames listed as SANs), configure individual per-node certificates with just the specific node's hostname.

The exact certificate size limit is hard to precisely specify, given the certificate is not sent as-is over the TCP/IP connection, however empirically certificates larger than 6kB can encounter this issue (as they can increase to 8kB in size when transmitted).