Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-2414

Connection terminated when packet exceeds NetworkStream buffer size

    XMLWordPrintable

Details

    • Task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0-beta.4
    • 3.0.0
    • library
    • None
    • 1

    Description

      Possible bug when running multiple (hundreds) of GetAsync requests in parallel...(we have a workaround so not urgent, but would be interested in feedback on the following):

      We have a wrapper for GetAsync that looks like this:

      public async Task<T> GetAsync(string key)
      {
         try {
            var result = await ICouchbaseCollectionObj.GetAsync(key);
            return result != null ? result.ContentAs<T>() : default(T);
         } catch () { ... }
      }
      

      I am seeing exceptions in the logs as follows...
      System.NullReferenceException: Object reference not set to an instance of an object.
      at Couchbase.KeyValue.GetResult.ParseSpecs()
      at Couchbase.KeyValue.GetResult.ContentAs[T]()

      Is there a way to check if the _contents of the result are null? Is this a bug? How is that we can we get a valid result back from GetAsync, but trying to then access the _contents of the result fails?

      Note that we only typically see this when we are calling GetAsync(key) in parallel for hundreds of keys by using a "GetMultiAsync" wrapper to our "GetAsync" wrapper as follows:

      public async Task<Dictionary<string,T>> GetMultiAsync<T>(string[] keys)
      {
         var result = new ConcurrentDictionary<string, T>();
         try
         {
            var GetAsyncTasks = keys.Select(async key =>
               { 
                  var document = await GetAsync<T>(key);
                  if (document != null) {
                     result.TryAdd(key, document);
                  }
               });
            await Task.WhenAll(GetAsyncTasks);
         } catch () { ... }
         return new Dictionary<string, T>(result);
      }
      

      When this occurs, the document for the key is not retrieved even though we can see from the CB UI that the document is there and is what we expect it to be.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jmorris Jeff Morris added a comment -

          This is easily reproducible; the size of the document exceeds the buffer size and the growth (simply doubling) algorithm doesn't work because now were using NetworkStream (instead of plain Sockets) and it terminates the connection as soon as its internal buffer is exceeded. The actual packet never even hits the network, thus you cannot see the request in a packet dump.

          jmorris Jeff Morris added a comment - This is easily reproducible; the size of the document exceeds the buffer size and the growth (simply doubling) algorithm doesn't work because now were using NetworkStream (instead of plain Sockets) and it terminates the connection as soon as its internal buffer is exceeded. The actual packet never even hits the network, thus you cannot see the request in a packet dump.
          jmorris Jeff Morris added a comment -

          Further looking into this shows that I can insert a document (JSONified string array) of
          20966470kb - which is just under the ~20mb max for Couchbase, but cannot do documents of size 20971590kb. This seems correct as what can be handled by CB, however, I do not get a VALUE_TOO_LARGE (0x0003) from the server - the connection is simply dropped or so it seems (wire shark isn't showing the request only the response on larger doc sizes probably due to size).

          jmorris Jeff Morris added a comment - Further looking into this shows that I can insert a document (JSONified string array) of 20966470kb - which is just under the ~20mb max for Couchbase, but cannot do documents of size 20971590kb. This seems correct as what can be handled by CB, however, I do not get a VALUE_TOO_LARGE (0x0003) from the server - the connection is simply dropped or so it seems (wire shark isn't showing the request only the response on larger doc sizes probably due to size).
          jmorris Jeff Morris added a comment - - edited

          Update:

          For documents that exceed ~20mb, but not by too much, the server rejects the document and returns a VALUE_TOO_LARGE (Ox0003) error from the server - the connection remains intact. For very large documents (~31MB), the NetworkStream buffer is exceeded which causes the socket to close - without sending the packet (my hunch, or because nothing is returned, but server could be nixing the connection).

          I believe it's the server; the fix for this ticket is too fail fast at docs exceeding 20MB

          Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

          jmorris Jeff Morris added a comment - - edited Update: For documents that exceed ~20mb, but not by too much, the server rejects the document and returns a VALUE_TOO_LARGE (Ox0003) error from the server - the connection remains intact. For very large documents (~31MB), the NetworkStream buffer is exceeded which causes the socket to close - without sending the packet (my hunch, or because nothing is returned, but server could be nixing the connection). I believe it's the server; the fix for this ticket is too fail fast at docs exceeding 20MB Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

          People

            jmorris Jeff Morris
            jmorris Jeff Morris
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty