Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-1267

SslConnection fails when two or more threads write using MuxIO

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: library
    • Labels:

      Description

      I believe this is a regression from NCBC-1260 - when SSL is enabled, a series of calls on multiple threads will cause a NotSupportedException to be thrown. The reason being that MuxIO shares the connection and SslStream is not thread safe.

      The breaking test is here: https://github.com/couchbase/couchbase-net-client/blob/master/Src/Couchbase.IntegrationTests/CouchbaseBucket_Ssl_Tests.cs#L80-L97

      2017-01-09 18:41:36,428 [23] DEBUG Couchbase.IO.Services.MultiplexingIOService - Endpoint: 10.142.160.101:11207 - 5e63123b-671a-40dd-b5ff-d0ab815890e2 System.AggregateException: One or more errors occurred. ---> System.NotSupportedException: The BeginWrite method cannot be called when another write operation is pending.
      at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
      at System.Net.Security._SslStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback asyncCallback, Object asyncState)
      at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_0(Stream stream, ReadWriteParameters args, AsyncCallback callback, Object state)
      at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
      at System.IO.Stream.BeginEndWriteAsync(Byte[] buffer, Int32 offset, Int32 count)
      at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
      at Couchbase.IO.SslConnection.<SendAsync>d__7.MoveNext() in C:\Users\jmorris\repos\couchbase-net-client\Src\Couchbase\IO\SslConnection.cs:line 191
      — End of inner exception stack trace —
      at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
      at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
      at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
      at Couchbase.IO.SslConnection.Send(Byte[] buffer) in C:\Users\jmorris\repos\couchbase-net-client\Src\Couchbase\IO\SslConnection.cs:line 175
      at Couchbase.IO.Services.MultiplexingIOService.Execute[T](IOperation`1 operation) in C:\Users\jmorris\repos\couchbase-net-client\Src\Couchbase\IO\Services\MultiplexingIOService.cs:line 103
      ---> (Inner Exception #0) System.NotSupportedException: The BeginWrite method cannot be called when another write operation is pending.
      at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
      at System.Net.Security._SslStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback asyncCallback, Object asyncState)
      at System.IO.Stream.<>c.<BeginEndWriteAsync>b__53_0(Stream stream, ReadWriteParameters args, AsyncCallback callback, Object state)
      at System.Threading.Tasks.TaskFactory`1.FromAsyncTrim[TInstance,TArgs](TInstance thisRef, TArgs args, Func`5 beginMethod, Func`3 endMethod)
      at System.IO.Stream.BeginEndWriteAsync(Byte[] buffer, Int32 offset, Int32 count)
      at System.IO.Stream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
      at Couchbase.IO.SslConnection.<SendAsync>d__7.MoveNext() in C:\Users\jmorris\repos\couchbase-net-client\Src\Couchbase\IO\SslConnection.cs:line 191<---

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          jmorris Jeff Morris added a comment -

          A possible work-around for GA may to not support TLS for 2.4.0...the solution is to write an Ssl version of MultiplexingConnection where writes are protected by a lock and reads are done on a separate read dedicated read thread.

          Show
          jmorris Jeff Morris added a comment - A possible work-around for GA may to not support TLS for 2.4.0...the solution is to write an Ssl version of MultiplexingConnection where writes are protected by a lock and reads are done on a separate read dedicated read thread.
          Hide
          jmorris Jeff Morris added a comment -

          Another work-around that came up in a conversation with Michael Goldsmith is to default to pooled IO if UseSll is enabled as a temporary fix. This would allow most users to get the benefits of using muxio and provide a work-around (documented) for SSL users. In the meantime, we could work on a solution for muxio-ssl.

          Show
          jmorris Jeff Morris added a comment - Another work-around that came up in a conversation with Michael Goldsmith is to default to pooled IO if UseSll is enabled as a temporary fix. This would allow most users to get the benefits of using muxio and provide a work-around (documented) for SSL users. In the meantime, we could work on a solution for muxio-ssl.

            People

            • Assignee:
              jmorris Jeff Morris
              Reporter:
              jmorris Jeff Morris
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.