Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
Regression in bootstrapping in 3.4.7 around NRE in PruneNodes
Description
When upgrading from 3.4.6 to 3.4.7 bootstrapping began to fail, showing this exception in the logs:
System.NullReferenceException: Object reference not set to an instance of an object. at Couchbase.Core.ClusterContext.<>c__DisplayClass70_0.<PruneNodes>b__1(IClusterNode x) at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at Couchbase.Core.ClusterContext.PruneNodes(BucketConfig config) at Couchbase.Core.ClusterContext.ProcessClusterMapAsync(BucketBase bucket, BucketConfig config)
Based on review of the stack trace, I believe this is caused by the lack of a null check on IClusterNode.Owner here:
Now it throws the exception instead of simply logging and moving on.
In case it's a factor, the Couchbase Server details are:
Couchbase Server 6.6.5 Enterprise
Running in K8S via the Autonomous Operator
Two buckets used by this service, one Couchbase and one Ephemeral
MDS scaling with separate services per node
Application details:
.NET 6 using ASP.NET Core MVC
Using DI extensions
Not using WaitUntilReadyAsync, simply requesting the bucket
Environment
None
Gerrit Reviews
None
Release Notes Description
None
Attachments
2
09 Jun 2023, 12:22 AM
08 Jun 2023, 11:36 PM
Activity
Show:
Jeffry Morris June 9, 2023 at 12:30 AM
@Brant Burnett -
Thanks for sharing the updates, I am working on a fix.
Brant Burnett June 9, 2023 at 12:23 AM
@Jeffry Morris
I've got the repro. The key is to setup your cluster with MDS and then bootstrap from a node that is not running the Data service. I'm not precisely certain why this is a repro of our production environment, since we only bootstrap from data nodes there. However, the same NRE occurs in PruneNodes.
Repro logs attached.
using Couchbase.Extensions.DependencyInjection;
using Couchbase.Extensions.Tracing.Otel.Tracing;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using TestApp;
var services = new ServiceCollection();
services
.AddLogging(builder =>
{
builder.SetMinimumLevel(LogLevel.Debug);
builder.AddConsole();
})
.AddCouchbase(options =>
{
options.ConnectionString = "couchbase://172.25.0.3:11210";
options.UserName = "Administrator";
options.Password = "password";
options.EnableDnsSrvResolution = false;
})
.AddCouchbaseBucket<IDefaultBucketProvider>("default");
var serviceProvider = services.BuildServiceProvider();
try
{
var bucketProvider = serviceProvider.GetRequiredService<IDefaultBucketProvider>();
var bucket = await bucketProvider.GetBucketAsync().ConfigureAwait(false);
var collection = bucket.DefaultCollection();
for (int i = 0; i < 1000; i++)
{
await collection.GetAsync("test").ConfigureAwait(false);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
Brant Burnett June 8, 2023 at 11:36 PM
@Jeffry Morris
Jeffry Morris June 8, 2023 at 11:17 PM
@Brant Burnett -
Would you be able to attach SDK logs?
Jeffry Morris June 8, 2023 at 3:13 PM
@Brant Burnett -
Thanks for the details, we will look into it!
Fixed
Pinned fields
Click on the next to a field label to start pinning.
When upgrading from 3.4.6 to 3.4.7 bootstrapping began to fail, showing this exception in the logs:
System.NullReferenceException: Object reference not set to an instance of an object. at Couchbase.Core.ClusterContext.<>c__DisplayClass70_0.<PruneNodes>b__1(IClusterNode x) at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext() at Couchbase.Core.ClusterContext.PruneNodes(BucketConfig config) at Couchbase.Core.ClusterContext.ProcessClusterMapAsync(BucketBase bucket, BucketConfig config)Based on review of the stack trace, I believe this is caused by the lack of a null check on IClusterNode.Owner here:
https://github.com/couchbase/couchbase-net-client/blob/b527bc98b681f89fae5954d7c1aad4eb01b49c35/src/Couchbase/Core/ClusterContext.cs#L834
I think this error probably existed before, but was being disguised until new error handling code was added in 3.4.7:
https://github.com/couchbase/couchbase-net-client/commit/7f3bf0f95142114b6453be6f4e26154fe884adbe#diff-ba5e9d3a81abb2d724c35296910c9b3a53782d6c9648fbcc3b4403cad8a60120R816
Now it throws the exception instead of simply logging and moving on.
In case it's a factor, the Couchbase Server details are:
Couchbase Server 6.6.5 Enterprise
Running in K8S via the Autonomous Operator
Two buckets used by this service, one Couchbase and one Ephemeral
MDS scaling with separate services per node
Application details:
.NET 6 using ASP.NET Core MVC
Using DI extensions
Not using WaitUntilReadyAsync, simply requesting the bucket