Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-2762

Threshold trace logging leaks memory

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.1.0
    • 3.1.1
    • library
    • None
    • 1

    Description

      See https://forums.couchbase.com/t/couchbase-net-sdk-memory-leak/28796/4

      I also have a memory dump file, but it's too large to attach.

      Attachments

        For Gerrit Dashboard: NCBC-2762
        # Subject Branch Project Status CR V

        Activity

          jmorris Jeff Morris added a comment -

          Brant Burnett- thanks for reporting!

          jmorris Jeff Morris added a comment - Brant Burnett - thanks for reporting!

          Brant Burnett, looks like the culprit is PropagateTagsUpwards.

           

          At the very least, it should verify that Activity.Parent is a CouchbaseNetClient Activity before adding the tags.

          richard.ponton Richard Ponton added a comment - Brant Burnett , looks like the culprit is PropagateTagsUpwards.   At the very least, it should verify that Activity.Parent is a CouchbaseNetClient Activity before adding the tags.

          That’s probably part of the issue, though I’m also surprised the HTTP activity is alive and not collected.

          btburnett3 Brant Burnett added a comment - That’s probably part of the issue, though I’m also surprised the HTTP activity is alive and not collected.

          Reproduction:

          using System;
          using System.Diagnostics;
          using System.Linq;
          using System.Threading.Tasks;
          using Couchbase.Extensions.DependencyInjection;
          using Microsoft.Extensions.DependencyInjection;
           
          var serviceProvider = new ServiceCollection()
              .AddLogging()
              .AddCouchbase(options =>
              {
                  options.ConnectionString = "couchbase://localhost";
                  options.ThresholdOptions!.Enabled = true;
                  options.UserName = "Administrator";
                  options.Password = "password";
              })
              .BuildServiceProvider();
           
          var tasks = Enumerable.Range(1, 100)
              .Select(i => Task.Run(async () =>
              {
                  for (var j = 1; j <= 1000; j++)
                  {
                      var activity = new Activity("MyTestActivity")
                          .AddTag("key", (i * j).ToString())
                          .Start();
           
                      var bucket = await serviceProvider.GetRequiredService<IBucketProvider>().GetBucketAsync("default");
                      var collection = bucket.DefaultCollection();
           
                      await collection.UpsertAsync($"doc-{i*j}", new {i = i*j});
           
                      activity.Stop();
                  }
              }));
           
          await Task.WhenAll(tasks);
          

          Note that when the connection pool scales up after 30 seconds of load, the activity from the original operation is the parent of the "initialize_connection" span. This is keeping the same activity alive for the entire life of the application. Combined that with PropagateTagsUpwards and the same Activity keeps getting more tags for every scale up operation or replacement of a dead connection. On top of that, Activity stores tags as a linked list, just pushing new tags in front. So as the same tag key is added repeatedly, it doesn't overwrite the previous value and instead adds a new tag to the linked list.

          btburnett3 Brant Burnett added a comment - Reproduction: using System; using System.Diagnostics; using System.Linq; using System.Threading.Tasks; using Couchbase.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;   var serviceProvider = new ServiceCollection() .AddLogging() .AddCouchbase(options => { options.ConnectionString = "couchbase://localhost" ; options.ThresholdOptions!.Enabled = true ; options.UserName = "Administrator" ; options.Password = "password" ; }) .BuildServiceProvider();   var tasks = Enumerable.Range(1, 100) .Select(i => Task.Run( async () => { for ( var j = 1; j <= 1000; j++) { var activity = new Activity( "MyTestActivity" ) .AddTag( "key" , (i * j).ToString()) .Start();   var bucket = await serviceProvider.GetRequiredService<IBucketProvider>().GetBucketAsync( "default" ); var collection = bucket.DefaultCollection();   await collection.UpsertAsync($ "doc-{i*j}" , new {i = i*j});   activity.Stop(); } }));   await Task.WhenAll(tasks); Note that when the connection pool scales up after 30 seconds of load, the activity from the original operation is the parent of the "initialize_connection" span. This is keeping the same activity alive for the entire life of the application. Combined that with PropagateTagsUpwards and the same Activity keeps getting more tags for every scale up operation or replacement of a dead connection. On top of that, Activity stores tags as a linked list, just pushing new tags in front. So as the same tag key is added repeatedly, it doesn't overwrite the previous value and instead adds a new tag to the linked list.

          People

            btburnett3 Brant Burnett
            btburnett3 Brant Burnett
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty