error message does not indicate certificate is the reason connection could not be made

One of our advantages with protostellar is that we should have more deterministic, clear error messages when connectivity does not work. This is not the case currently.

I was running an environment where the cert is not trusted.

When running the client, I saw only this error message for a set of code that connects with "couchbase2://" and then does a waitUntilReady, logging at info level:

/Users/ingenthr/Library/Java/JavaVirtualMachines/temurin-17.0.9/Contents/Home/bin/java -javaagent:/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar=50185:/Applications/IntelliJ IDEA.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/ingenthr/src/cngateway-demo-client/target/classes:/Users/ingenthr/.m2/repository/com/couchbase/client/java-client/3.5.2/java-client-3.5.2.jar:/Users/ingenthr/.m2/repository/com/couchbase/client/core-io/2.5.2/core-io-2.5.2.jar:/Users/ingenthr/.m2/repository/io/projectreactor/reactor-core/3.5.8/reactor-core-3.5.8.jar:/Users/ingenthr/.m2/repository/org/reactivestreams/reactive-streams/1.0.4/reactive-streams-1.0.4.jar:/Users/ingenthr/.m2/repository/ch/qos/logback/logback-classic/1.4.7/logback-classic-1.4.7.jar:/Users/ingenthr/.m2/repository/ch/qos/logback/logback-core/1.4.7/logback-core-1.4.7.jar:/Users/ingenthr/.m2/repository/org/slf4j/slf4j-api/2.0.4/slf4j-api-2.0.4.jar:/Users/ingenthr/.m2/repository/io/opentelemetry/opentelemetry-api/1.31.0/opentelemetry-api-1.31.0.jar:/Users/ingenthr/.m2/repository/io/opentelemetry/opentelemetry-context/1.31.0/opentelemetry-context-1.31.0.jar com.couchbase.sdk.demo.Main
{{2024-01-12T18:49:27,658Z [main] INFO com.couchbase.core - [CoreCreatedEvent]

{"clientVersion":"3.5.2","clientGitHash":"${buildNumber}","coreVersion":"2.5.2","coreGitHash":"${buildNumber}","userAgent":"couchbase-java/3.5.2 (Mac OS X 13.6.3 aarch64; OpenJDK 64-Bit Server VM 17.0.9+9)","maxNumRequestsInRetry":32768,"ioEnvironment":

,"ioConfig":\\{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","eventingCircuitBreakerConfig":"disabled","backupCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":1000,"configIdleRedialTimeoutMs":300000,"memcachedHashingStrategy":"StandardMemcachedHashingStrategy","timerConfig":{"numTimers":1,"tickDurationMs":10,"numBuckets":512}},"compressionConfig":

,"securityConfig":

,"timeoutConfig":

,"loggerConfig":

,"orphanReporterConfig":

,"thresholdLoggingTracerConfig":

,"loggingMeterConfig":

,"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdLoggingTracer","meter":"LoggingMeter","numRequestCallbacks":0,"scheduler":"ParallelScheduler","schedulerThreadCount":10,"transactionsConfig":
{"durabilityLevel":"MAJORITY","timeoutMs":15000,"cleanupConfig":

,"numAtrs":1024,"metadataCollection":"none","scanConsistency":"none"}}}}
Exception in thread "main" com.couchbase.client.core.error.UnambiguousTimeoutException: Timed out while waiting for Protostellar endpoint localhost:18098 {}
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.Process finished with exit code 1

But the problem actually isn't a timeout. If I turn the log level up to debug (which I shouldn't need to do to see the issue), I see this at the end:

…some repetition elided…
2024-01-12T18:50:58,162Z [grpc-default-executor-0] DEBUG c.c.c.c.d.i.n.h.s.ReferenceCountedOpenSslContext - verification of certificate failed
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at com.couchbase.client.core.deps.io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:790)
at com.couchbase.client.core.deps.io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36)
at com.couchbase.client.core.deps.io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48)
at com.couchbase.client.core.deps.io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1534)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:96)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1509)
at com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler$SslTasksRunner.run(SslHandler.java:1889)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 17 common frames omitted
2024-01-12T18:50:58,180Z [grpc-default-executor-0] DEBUG c.c.c.c.d.i.n.h.s.ReferenceCountedOpenSslContext - verification of certificate failed
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at com.couchbase.client.core.deps.io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:790)
at com.couchbase.client.core.deps.io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36)
at com.couchbase.client.core.deps.io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48)
at com.couchbase.client.core.deps.io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1534)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:96)
at com.couchbase.client.core.deps.io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1509)
at com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler$SslTasksRunner.run(SslHandler.java:1889)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 17 common frames omitted
Exception in thread "main" com.couchbase.client.core.error.UnambiguousTimeoutException: Timed out while waiting for Protostellar endpoint localhost:18098 {}
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.Process finished with exit code 1

It looks like there are multiple tries with an untrusted cert before giving up, but the only error I receive at INFO level is a timeout.

As configuration of certificates with protostellar is complicated, this is a likely failure case.

I will note that to override this particular setting and insecurely force it to trust the self-signed CNG, I am able to configure it like this:

        Cluster cluster = Cluster.connect(

                connectionString,

                ClusterOptions.clusterOptions(username, password).environment(env -> {

                    env.securityConfig(sec -> sec.enableCertificateVerification(false));

//                    env.securityConfig(SecurityConfig.trustCertificate(Paths.get("/opt/homebrew/etc/pki/ca.crt")));  // the proper way to add a trusted cert

                    env.thresholdLoggingTracerConfig().kvThreshold(Duration.ofMillis(1));

                })

        );

        // Get a bucket reference

        bucket = cluster.bucket(bucketName);

        bucket.waitUntilReady(Duration.ofSeconds(10));