Subdoc failures after restarting CB server

-
Patchset 3 has fixed it! Running the whole suite now to check for regressions, and I'll resolve the ticket once that is done.

With one thread: http://sdkqe-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-6.5.1-6299/SvcRestartAll-SUBDOC/06-16-20/072953/fd9e0950709e31920a254a6b1fd340f0-SD.html
Logs:

Also included the Jenkins console log as it has stack traces:

Running with amended patch:

http://sdkqe-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-6.5.1-6299/SvcRestartAll-SUBDOC/06-16-20/071012/f601b9632c46ef2de95dbc310077152d-SD.html

:

Running with http://review.couchbase.org/c/couchbase-net-client/+/130602/1 , I see there is a CircuitBreakerException happening now.

Graph: http://sdkqe-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-6.0.4-3082/SvcRestartAll-SUBDOC/06-16-20/043638/5ba23646c7bbb59f8908099e49a646d8-SD.html

Logs:

Also, here's some of the Sdkd stack traces that are being thrown:

I think I have 'fixed' the issue running with 6.5 by just increasing the WaitForReady to 20s - though I'm still running tests for it now. UPDATE: Still getting timeouts with 20s.

But for CB <6.5 I can't use this. Looking at the logs I think the issue is that the GetClusterConfig is returning BucketNotConnected (occasionally), so connecting to the cluster is failing. Filed - .