Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57002

TLS handshake fails with "ssl_get_prev_session:session id context uninitialized}" when using Session Tickets + Require Client Certificates

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 0
    • Unknown

    Description

      Issue Resolution
      When using .NET SDK on Windows 10 client and client certs were enabled on CB Server, the Data-Service did not establish a connection and client bootstrap failed with a OpenSSL “session id context uninitialized" error. Data-Service has been updated to disable TLS session resume.

       

      Summary

      If an SDK client connection attempts to negotiate a TLS connection, and the client supports TLS Session Resume (RFC 5077) client certs, and client certs are enabled (Security -> "Require Client Certificate" -> Enabled), then KV-Engine fails to establish the connection and client bootstrap fails with the following OpenSSL error:

      2023-05-18T13:08:23.069545+01:00 WARNING 68: Unrecoverable error encountered: ["reading","error"], ssl_error: [{error:00000001:lib(0):func(0):reason(1)},{error:140D9115:SSL routines:ssl_get_prev_session:session id context uninitialized}], shutting down connection
      

      Such Couchbase clients include .NET 6 applications running on Windows 10 Pro, as that automatically enables TLS Session Resume.

      Workaround

      On Windows, use SChannel in TLSv1.3 mode, which does not appear to support TLS Session Resume (pre_shared_key client Hello extension):

      1. Modify the ClusterOptions passed to ConnectAsync to add EnabledSslProtocols = SslProtocols.None, which tells the OS to use the highest available TLS version. (Note you cannot explicitly specify SslProtocols.Tls13, as that fails to handshake. This is somewhat counter-intuitive...)

                var clusterOptions = new ClusterOptions{
                    // other options, e.g. 
                    KvIgnoreRemoteCertificateNameMismatch = true, // development only. do not include in any production configuration
        ...
                    EnabledSslProtocols = SslProtocols.None
                }
                .WithConnectionString("couchbases://"+connection_string)
                .WithCredentials(username: userid, password: password)
                .WithLogging(LoggerFactory.Create(builder => { builder.AddFilter("Couchbase", LogLevel.Debug).AddConsole(); }));
         
                var cluster = await Couchbase.Cluster.ConnectAsync(
                    clusterOptions)
                .ConfigureAwait(false);
        

      2. On Windows 10 Pro (22H2), TLSv1.3 is not enabled by default at the OS level, so must be turned on. This requires modifying the registry by adding the following keys:

        Windows Registry Editor Version 5.00
        [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
        "DisabledByDefault"=dword:00000000
        "Enabled"=dword:00000001
        

        The simplest way to do this is to create a file named "tls13_enable.reg", copy the contents above into it, then run it.

      To undo this change disable TLSv1.3 support in the OS, create a file named "tls13_disable.reg" containing the following contents, then run it:

      Windows Registry Editor Version 5.00
      [-HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3]
      

      Steps to Reproduce

      1. Setup a cluster with 3 buckets named:
        • logs
        • render
        • renderaudio
      2. From the UI, Security -> "Certificates" tab -> "Require Client Certificate" -> Enable.
      3. On a Windows 10 machine, clone https://github.com/bradflood/couchbase-dotnet-sample (on Windows it can be easier to download that as a zip file.
      4. Install .NET SDK 6 from Microsoft.
      5. Open Command Prompt, set following 3 env vars:

        set cb_connection_string=<HOSTNAME>
        set cb_userid=<USER>
        set cb_password=<PASSWORD>
        

      6. cd App1
        dotnet run > run.out (make sure the stdout logs are not being truncated, for example when run in an IDE terminal window)

      Expected behavour
      Test program runs and terminates cleanly.

      Actual behaviour
      Test program fails with an exception similar to:

      [17:48:38 DBG] Waiting for 00:00:02.5000000 before polling.
      [17:48:38 INF] Successfully retrieved DNS SRV entries: [svc-dqis-node-032.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207,svc-dqis-node-033.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207,svc-dqis-node-036.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207]
      [17:48:38 DBG] Bootstrapping: global bootstrapping with node svc-dqis-node-032.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207
      [17:48:38 INF] Using the ChannelConnectionPool.
      [17:48:41 DBG] Done waiting, polling...
      [17:48:41 INF] Bootstrapping: The handler can no longer connect  to the cluster and will attempt to rebootstrap against the DNS SRV records.
      [17:48:41 INF] Successfully retrieved DNS SRV entries: [svc-dqis-node-032.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207,svc-dqis-node-033.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207,svc-dqis-node-036.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207]
      [17:48:41 DBG] Bootstrapping: global bootstrapping with node svc-dqis-node-032.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207
      [17:48:41 INF] Using the ChannelConnectionPool.
      [17:48:48 DBG] Bootstrapping: attempted global bootstrapping on endpoint svc-dqis-node-032.zdgtdtmt9pjmpd5w.cloud.couchbase.com:11207 has failed.
      System.Net.Sockets.SocketException (60): Operation timed out
         at Couchbase.Core.IO.Connections.ConnectionFactory.CreateAndConnectAsync(HostEndpointWithPort hostEndpoint, CancellationToken cancellationToken) in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/IO/Connections/ConnectionFactory.cs:line 68
         at Couchbase.Core.IO.Connections.ConnectionPoolBase.CreateConnectionAsync(CancellationToken cancellationToken) in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/IO/Connections/ConnectionPoolBase.cs:line 84
         at Couchbase.Core.IO.Connections.Channels.ChannelConnectionPool.<>c__DisplayClass30_0.<<AddConnectionsAsync>g__StartConnection|0>d.MoveNext() in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/IO/Connections/Channels/ChannelConnectionPool.cs:line 277
      --- End of stack trace from previous location ---
         at Couchbase.Core.IO.Connections.Channels.ChannelConnectionPool.InitializeAsync(CancellationToken cancellationToken) in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/IO/Connections/Channels/ChannelConnectionPool.cs:line 102
         at Couchbase.Core.ClusterNode.InitializeAsync() in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/ClusterNode.cs:line 226
         at Couchbase.Core.DI.ClusterNodeFactory.CreateAndConnectAsync(HostEndpointWithPort endPoint, NodeAdapter nodeAdapter, CancellationToken cancellationToken) in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/DI/ClusterNodeFactory.cs:line 65
         at Couchbase.Core.ClusterContext.BootstrapGlobalAsync() in /Users/jeffry.morris/Documents/source/couchbase-net-client/src/Couchbase/Core/ClusterContext.cs:line 366
      

      Additionally, memcached shows an error similar to:

      2023-05-18T13:08:23.069545+01:00 WARNING 68: Unrecoverable error encountered: ["reading","error"], ssl_error: [{error:00000001:lib(0):func(0):reason(1)},{error:140D9115:SSL routines:ssl_get_prev_session:session id context uninitialized}], shutting down connection
      

      Alternatively, one can perform a basic test with OpenSSL s_cilent:

      • Same environment setup (cluster with "Require Client Certificate" set to Enable.
      • Establish an initial connection, save the session, then attempt to re-use the session for a second connection:

        $ echo | install/bin/openssl s_client -connect 127.0.0.1:11998 -servername example.com -sess_out session -tls1_3
        $ echo | install/bin/openssl s_client -connect 127.0.0.1:11998 -servername example.com -tls1_3 -sess_in session | rg '(New|Reused)'
        

        This should run without error, printing "Reused" for the second connection attempt. However it currently fails with the same alert error as seen via .NET, and connection fails the handshake:

        8670508544:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1544:SSL alert number 80
        New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
        

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              will.broadbelt Will Broadbelt
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty