Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
None
-
None
-
MacOS 12.3, .NET 6, Console App
-
1
-
High
Description
I wrote a performance testing tool to test how many IOPS the .NET SDK supports via the TPL library. The repo can be found here:
https://github.com/biozal/cb-dotnet-load-test
When increasing the KeepConstant threads beyond 1500 I get an exception fault:
10:15:39 [INF] NBomber '2.1.5' Started a new session: '2022-03-16_14.15.58_session_e7a6206f'.
10:15:39 [INF] NBomber started as single node.
10:15:39 [INF] Plugins: no plugins were loaded.
10:15:39 [INF] Reporting sinks: no reporting sinks were loaded.
10:15:39 [INF] Starting init...
10:15:39 [INF] Target scenarios: 'Test Couchbase'.
10:15:39 [INF] Start init scenario: 'Test Couchbase'.
10:15:51 [INF] Init finished.
10:15:51 [INF] Starting warm up...
10:16:02 [ERR] The operation /739629 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739631 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739624 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739630 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739625 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739628 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739622 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739627 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739632 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] The operation /739623 timed out after 00:00:02.5000000. It was retried 10 times using
Couchbase.Core.Retry.BestEffortRetryStrategy. - at Couchbase.Utils.ThrowHelper.ThrowTimeoutException(IOperation operation, IErrorContext context)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:02 [ERR] Cannot access a disposed object.
Object name: 'DataFlowConnectionPool'. - at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:03 [ERR] Cannot access a disposed object.
Object name: 'DataFlowConnectionPool'. - at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:06 [ERR] Cannot access a disposed object.
Object name: 'DataFlowConnectionPool'. - at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
10:16:08 [ERR] Cannot access a disposed object.
Object name: 'DataFlowConnectionPool'. - at Couchbase.Utils.ThrowHelper.ThrowObjectDisposedException(String objectName)
at Couchbase.Core.IO.Connections.DataFlow.DataFlowConnectionPool.SendAsync(IOperation operation, CancellationToken cancellationToken)
at Couchbase.Core.ClusterNode.<>c.<ExecuteOp>b__127_0(IOperation op2, Object state, CancellationToken effectiveToken)
at Couchbase.Core.ClusterNode.ExecuteOp(Func`4 sender, IOperation op, Object state, CancellationTokenPair tokenPair)
at Couchbase.Core.ClusterNode.SendAsyncWithCircuitBreaker(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.CouchbaseBucket.SendAsync(IOperation op, CancellationTokenPair tokenPair)
at Couchbase.Core.Retry.RetryOrchestrator.RetryAsync(BucketBase bucket, IOperation operation, CancellationTokenPair tokenPair)
at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)
at Program.<<Main>$>g__PopuplateDatabase|0_2(String json, ICouchbaseCollection collection) in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 80
at Program.<>c_DisplayClass0_0.<<<Main>$>b_0>d.MoveNext() in
/Users/labeaaa/Developer/couchbase/cb-dotnet-load-test/src/cb-dotnet-load-test/Program.cs:line 40
Once I get this exception fault, it's not recoverable without completely disposing the Bucket and Cluster object and opening a new connection.
Reproduce Steps:
1. Clone Repo from here: https://github.com/biozal/cb-dotnet-load-test
2. Go into Program.cs and change the Simulation KeepConstant copies value from the default to a higher number, like 2000.
3. dotnet build
4. dotnet run
You should get the exception fault shown. TPL is the better way to handle the load as the standard async library has overhead involved with it and clients looking for performance and using all the hardware at their disposal will be using this library, which is a standard way of doing Parallel tasks in .NET. I also tried just a quick test using Parellel For loop and if I push my MaxDegreeOfParallelism beyond 2 I see the same exception fault.
cc: Denis Rosa, Roi Katz