Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-3161

Exception Fault using TPL or performance testing tool (NBomber)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 3.2.9
    • None
    • None
    • None
    • MacOS 12.3, .NET 6, Console App
    • 1
    • High

    Description

      I wrote a performance testing tool to test how many IOPS the .NET SDK supports via the TPL library.  The repo can be found here:
      https://github.com/biozal/cb-dotnet-load-test

      When increasing the KeepConstant threads beyond 1500 I get an exception fault:
                                                                                       
      10:15:39 [INF] NBomber '2.1.5' Started a new session: '2022-03-16_14.15.58_session_e7a6206f'.
      10:15:39 [INF] NBomber started as single node.
      10:15:39 [INF] Plugins: no plugins were loaded.
      10:15:39 [INF] Reporting sinks: no reporting sinks were loaded.
      10:15:39 [INF] Starting init...
      10:15:39 [INF] Target scenarios: 'Test Couchbase'.
      10:15:39 [INF] Start init scenario: 'Test Couchbase'.
      10:15:51 [INF] Init finished.
      10:15:51 [INF] Starting warm up...
      10:16:02 [ERR] The operation /739629 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739631 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739624 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739630 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739625 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739628 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739622 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739627 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739632 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] The operation /739623 timed out after 00:00:02.5000000. It was retried 10 times using 
      Couchbase.Core.Retry.BestEffortRetryStrategy. -    at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:02 [ERR] Cannot access a disposed object.                                          
      Object name: 'DataFlowConnectionPool'. -    at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
         at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
         at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
         at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
         at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:03 [ERR] Cannot access a disposed object.                                           
      Object name: 'DataFlowConnectionPool'. -    at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
         at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
         at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
         at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
         at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:06 [ERR] Cannot access a disposed object.                                           
      Object name: 'DataFlowConnectionPool'. -    at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
         at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
         at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
         at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
         at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
      10:16:08 [ERR] Cannot access a disposed object.                                           
      Object name: 'DataFlowConnectionPool'. -    at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
         at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
         at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
         at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
         at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
         at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
         at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
         at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
         at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in 
      /Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40

       

      Once I get this exception fault, it's not recoverable without completely disposing the Bucket and Cluster object and opening a new connection.  

      Reproduce Steps:
      1.  Clone Repo from here:  https://github.com/biozal/cb-dotnet-load-test
      2.  Go into Program.cs and change the Simulation KeepConstant copies value from the default to a higher number, like 2000.  
      3.  dotnet build
      4.  dotnet run

      You should get the exception fault shown.  TPL is the better way to handle the load as the standard async library has overhead involved with it and clients looking for performance and using all the hardware at their disposal will be using this library, which is a standard way of doing Parallel tasks in .NET.  I also tried just a quick test using Parellel For loop and if I push my MaxDegreeOfParallelism beyond 2 I see the same exception fault.

      cc:  Denis Rosa, Roi Katz 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jmorris Jeff Morris
            aaron.labeau Aaron LaBeau
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty