OperationCanceledException continues after rebalance completes

Initial discussion on the issue: https://forums.couchbase.com/t/couchbase-v3-sdk-kvnotmyvbucket-errors-after-add-node-rebalance/35438

Initial ticket for the issue: https://issues.couchbase.com/browse/NCBC-3350

Preconditions:

1. Have a cluster of 2 nodes where one get traffic (call it #1) and node waiting to be added to the cluster via rebalancing (call it #2)
2. Have a .NET client sending requests to the cluster

Steps

1. Block connections from the .NET client to the node #2
2. Start rebalance operation
3. In the middle of rebalance unblock connections to the node #2
4. Wait for the rebalance to finish

Expected Result

The app is recovered after rebalance is finished and continue working properly

Actual Result

The app is not recovered and constantly throwing exceptions:

• Timeouts exceptions before v3.4.5
• TaskCancelledExceptions after v3.4.5 (this is not fixed in v3.4.6 which can be verified with this test)

Dev Notes

Lots of investigation details can be found on the forum topic and in the previous ticket (the links are above).

The latest details:

There is the difference between this patch set (that solves the issue) https://review.couchbase.org/c/couchbase-net-client/+/186991 and the released v3.4.5 (and v3.4.6) version.

It is in the ClusterContext.ProcessClusterMap method. In the patch and I believe in all previous versions it propagated all exceptions that might occur during connecting to the bucket class, but in the v3.4.5 there is a try catch block that handles all exceptions and just log them. This make the CouchbaseBucket.ConfigUpdatedAsync method to think that connection was established and update its CurrentConfig prop (which prevents from updating it in the future), but in the |Nodes collection there is still no new node.

Current code that hides exceptions: https://github.com/couchbase/couchbase-net-client/blob/master/src/Couchbase/Core/ClusterContext.cs#L813-L816