Uploaded image for project: 'Java Couchbase JVM Core'
  1. Java Couchbase JVM Core
  2. JVMCBC-654

Java SDK unable to connect to externally exposed operator-operated cluster

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.7.4
    • 1.7.7
    • Core
    • None
    • 1

    Description

      The Java SDK cannot connect to a cluster running in Kubernetes exposed via LoadBalancers (using AlternateAddress feature).

      The stack trace is as follows:

      objc[59177]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/bin/java (0x10483d4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x1058ac4e0). One of the two will be used. Which one is undefined.
      Exception in thread "main" com.couchbase.client.core.config.ConfigurationException: Could not open bucket.
      	at com.couchbase.client.core.config.DefaultConfigurationProvider$6.call(DefaultConfigurationProvider.java:304)
      	at com.couchbase.client.core.config.DefaultConfigurationProvider$6.call(DefaultConfigurationProvider.java:301)
       
      ...
       
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
      	at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
      	at rx.Subscriber.setProducer(Subscriber.java:211)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
      	at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:103)
      	at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java:508)
      	at com.couchbase.client.core.endpoint.AbstractGenericHandler.access$000(AbstractGenericHandler.java:86)
      	at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:526)
      	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: com.couchbase.client.core.CouchbaseException: Could not parse configuration
      	at com.couchbase.client.core.config.parser.BucketConfigParser.parse(BucketConfigParser.java:55)
      	at com.couchbase.client.core.config.loader.AbstractLoader$1.call(AbstractLoader.java:204)
      	at com.couchbase.client.core.config.loader.AbstractLoader$1.call(AbstractLoader.java:200)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:69)
      	... 24 more
      Caused by: com.couchbase.client.deps.com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.couchbase.client.core.config.DefaultNodeInfo`, problem: Could not analyze hostname from config.
       at [Source: (String)"{"rev":102,"name":"default","uri":"/pools/default/buckets/default?bucket_uuid=020008d270ad55afb27eeaae157abb75","streamingUri":"/pools/default/bucketsStreaming/default?bucket_uuid=020008d270ad55afb27eeaae157abb75","nodes":[{"couchApiBase":"http://cb-example-0000.cb-example.default.svc:8092/default%2B020008d270ad55afb27eeaae157abb75","hostname":"cb-example-0000.cb-example.default.svc:8091","ports":{"proxy":11211,"direct":11210},"alternateAddresses":{"external":{"hostname":"cb-example-0000.cb-exam"[truncated 8992 chars]; line: 1, column: 2881] (through reference chain: com.couchbase.client.core.config.DefaultCouchbaseBucketConfig["nodes"]->java.util.ArrayList[0])
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1608)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:484)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:503)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:285)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:229)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:195)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:422)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:286)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:27)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:530)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:194)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:161)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1611)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1219)
      	at com.couchbase.client.core.config.parser.BucketConfigParser.parse(BucketConfigParser.java:53)
      	... 27 more
      Caused by: com.couchbase.client.core.CouchbaseException: Could not analyze hostname from config.
      	at com.couchbase.client.core.config.DefaultNodeInfo.<init>(DefaultNodeInfo.java:72)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124)
      	at com.couchbase.client.deps.com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:283)
      	... 50 more
      Caused by: java.lang.IllegalArgumentException: Could not create NetworkAddress.
      	at com.couchbase.client.core.utils.NetworkAddress.<init>(NetworkAddress.java:88)
      	at com.couchbase.client.core.utils.NetworkAddress.<init>(NetworkAddress.java:93)
      	at com.couchbase.client.core.utils.NetworkAddress.create(NetworkAddress.java:100)
      	at com.couchbase.client.core.config.DefaultNodeInfo.<init>(DefaultNodeInfo.java:70)
      	... 56 more
      Caused by: java.net.UnknownHostException: cb-example-0000.cb-example.default.svc
      	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
      	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
      	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
      	at com.couchbase.client.core.utils.NetworkAddress.<init>(NetworkAddress.java:70)
      	... 59 more
      Caused by: rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: {"rev":102,"name":"default","uri":"/pools/default/buckets/default?bucket_uuid=020008d270ad55afb27eeaae157abb75","streamingUri":"/pools/default/bucketsStreaming/default?bucket_uuid=020008d270ad55afb27eeaae157abb75","nodes":[{"couchApiBase":"http://cb-example-0000.cb-example.default.svc:8092/default%2B020008d270ad55afb27eeaae157abb75","hostname":"cb-example-0000.cb-example.default.svc:8091","ports":{"proxy":11211,"direct":11210},"alternateAddresses":{"external":{"hostname":"cb-example-0000.cb-example.marabine.co.uk"}}},{"couchApiBase":"http://cb-example-0001.cb-example.default.svc:8092/default%2B020008d270ad55afb27eeaae157abb75","hostname":"cb-example-0001.cb-example.default.svc:8091","ports":{"proxy":11211,"direct":11210},"alternateAddresses":{"external":{"hostname":"cb-example-0001.cb-example.marabine.co.uk"}}},{"couchApiBase":"http://cb-example-0002.cb-example.default.svc:8092/default%2B020008d270ad55afb27eeaae157abb75","hostname":"cb-example-0002.cb-example.default.svc:8091","ports":{"proxy":11211,"direct":11210},"alternateAddresses":{"external":{"hostname":"cb-example-0002.cb-example.marabine.co.uk"}}}],"nodesExt":[{"services":{"mgmt":8091,"mgmtSSL":18091,"cbas":8095,"cbasCc":9111,"cbasAdmin":9110,"cbasSSL":18095,"eventingAdminPort":8096,"eventingSSL":18096,"fts":8094,"ftsSSL":18094,"indexAdmin":9100,"indexScan":9101,"indexHttp":9102,"indexStreamInit":9103,"indexStreamCatchup":9104,"indexStreamMaint":9105,"indexHttps":19102,"capiSSL":18092,"capi":8092,"kvSSL":11207,"projector":9999,"kv":11210,"moxi":11211,"n1ql":8093,"n1qlSSL":18093},"thisNode":true,"hostname":"cb-example-0000.cb-example.default.svc","alternateAddresses":{"external":{"hostname":"cb-example-0000.cb-example.marabine.co.uk"}}},{"services":{"mgmt":8091,"mgmtSSL":18091,"cbas":8095,"cbasCc":9111,"cbasAdmin":9110,"cbasSSL":18095,"eventingAdminPort":8096,"eventingSSL":18096,"fts":8094,"ftsSSL":18094,"indexAdmin":9100,"indexScan":9101,"indexHttp":9102,"indexStreamInit":9103,"indexStreamCatchup":9104,"indexStreamMaint":9105,"indexHttps":19102,"capiSSL":18092,"capi":8092,"kvSSL":11207,"projector":9999,"kv":11210,"moxi":11211,"n1ql":8093,"n1qlSSL":18093},"hostname":"cb-example-0001.cb-example.default.svc","alternateAddresses":{"external":{"hostname":"cb-example-0001.cb-example.marabine.co.uk"}}},{"services":{"mgmt":8091,"mgmtSSL":18091,"cbas":8095,"cbasCc":9111,"cbasAdmin":9110,"cbasSSL":18095,"eventingAdminPort":8096,"eventingSSL":18096,"fts":8094,"ftsSSL":18094,"indexAdmin":9100,"indexScan":9101,"indexHttp":9102,"indexStreamInit":9103,"indexStreamCatchup":9104,"indexStreamMaint":9105,"indexHttps":19102,"capiSSL":18092,"capi":8092,"kvSSL":11207,"projector":9999,"kv":11210,"moxi":11211,"n1ql":8093,"n1qlSSL":18093},"hostname":"cb-example-0002.cb-example.default.svc","alternateAddresses":{"external":{"hostname":"cb-example-0002.cb-example.marabine.co.uk"}}}],"nodeLocator":"vbucket","uuid":"020008d270ad55afb27eeaae157abb75","ddocs":{"uri":"/pools/default/buckets/default/ddocs"},"vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["cb-example-0000.cb-example.default.svc:11210","cb-example-0001.cb-example.default.svc:11210","cb-example-0002.cb-example.default.svc:11210"],"vBucketMap":[[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1]]},"bucketCapabilitiesVer":"","bucketCapabilities":["couchapi","xattr","dcp","cbhello","touch","cccp","xdcrCheckpointing","nodesExt"]}
      	at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73)
      	... 24 more
      

      The key section is:

      Caused by: java.net.UnknownHostException: cb-example-0000.cb-example.default.svc
      

      It seems the SDK tries to resolve the original hostname of each node, even if (as in this case and in almost every case running in Kubernetes) that's not possible.

      To workaround this I can put duff entries in my /etc/hosts file so that at least the hostnames resolve:

      192.168.245.245 cb-example-0000.cb-example.default.svc
      192.168.245.246 cb-example-0001.cb-example.default.svc
      192.168.245.247 cb-example-0002.cb-example.default.svc
      

      Unfortunately, this still fails, with a different error:

      Exception in thread "main" com.couchbase.client.core.CouchbaseException: java.util.NoSuchElementException: Sequence contains no elements
      	at com.couchbase.client.java.CouchbaseAsyncCluster$OpenBucketErrorHandler.call(CouchbaseAsyncCluster.java:692)
      	at com.couchbase.client.java.CouchbaseAsyncCluster$OpenBucketErrorHandler.call(CouchbaseAsyncCluster.java:665)
      	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)
      	at rx.observers.Subscribers$5.onError(Subscribers.java:230)
      	at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.checkTerminated(OperatorObserveOn.java:273)
      	at rx.internal.operators.OperatorObserveOn$ObserveOnSubscriber.call(OperatorObserveOn.java:216)
      	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.util.NoSuchElementException: Sequence contains no elements
      	at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSingle.java:115)
      	at rx.internal.operators.DeferredScalarSubscriber.complete(DeferredScalarSubscriber.java:83)
      	at rx.internal.operators.OnSubscribeTakeLastOne$TakeLastOneSubscriber.onCompleted(OnSubscribeTakeLastOne.java:55)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:656)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onCompleted(OperatorMerge.java:281)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onCompleted(OnSubscribeMap.java:97)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:656)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onCompleted(OperatorMerge.java:281)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onCompleted(OnSubscribeMap.java:97)
      	at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:285)
      	at rx.Subscriber.setProducer(Subscriber.java:211)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
      	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:233)
      	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
      	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
      	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
      	at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.fastPath(OnSubscribeFromIterable.java:173)
      	at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:86)
      	at rx.Subscriber.setProducer(Subscriber.java:211)
      	at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:63)
      	at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:34)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
      	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
      	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
      	at rx.observers.Subscribers$5.onNext(Subscribers.java:235)
      	at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.fastPath(OnSubscribeFromIterable.java:173)
      	at rx.internal.operators.OnSubscribeFromIterable$IterableProducer.request(OnSubscribeFromIterable.java:86)
      	at rx.Subscriber.setProducer(Subscriber.java:211)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
      	at rx.Subscriber.setProducer(Subscriber.java:205)
      	at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:63)
      	at rx.internal.operators.OnSubscribeFromIterable.call(OnSubscribeFromIterable.java:34)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:235)
      	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
      	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
      	at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
      	at rx.Observable.unsafeSubscribe(Observable.java:10327)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
      	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onNext(OperatorOnErrorResumeNextViaFunction.java:154)
      	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onNext(OnSubscribeDoOnEach.java:101)
      	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onNext(OnSubscribeDoOnEach.java:101)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
      	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onNext(OnSubscribeDoOnEach.java:101)
      	at rx.internal.operators.OperatorTake$1.onNext(OperatorTake.java:79)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355)
      	at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846)
      	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onNext(OperatorOnErrorResumeNextViaFunction.java:154)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.emitScalar(OperatorMerge.java:395)
      	at rx.internal.operators.OperatorMerge$MergeSubscriber.tryEmit(OperatorMerge.java:355)
      	at rx.internal.operators.OperatorMerge$InnerSubscriber.onNext(OperatorMerge.java:846)
      	at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
      	at com.couchbase.client.core.utils.Buffers$2$1.onNext(Buffers.java:84)
      	at rx.observers.SafeSubscriber.onNext(SafeSubscriber.java:134)
      	at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
      	at rx.Subscriber.setProducer(Subscriber.java:209)
      	at rx.Subscriber.setProducer(Subscriber.java:205)
      	at rx.subjects.AsyncSubject.onCompleted(AsyncSubject.java:103)
      	at com.couchbase.client.core.endpoint.AbstractGenericHandler.completeResponse(AbstractGenericHandler.java:508)
      	at com.couchbase.client.core.endpoint.AbstractGenericHandler.access$000(AbstractGenericHandler.java:86)
      	at com.couchbase.client.core.endpoint.AbstractGenericHandler$1.call(AbstractGenericHandler.java:526)
      	... 8 more
      

      From digging around in the source code, I noticed that the AlternateAddress stuff had been changed a lot in https://github.com/couchbase/couchbase-jvm-core/commit/35de201b0d2f510f606e2eb99302db55c54af876, introduced in 2.6.1 (1.6.1 core).

      So I tried using 2.6.0 Java SDK and it worked correctly (after adding the /etc/hosts entry, no version of the Java SDK works without that change), which means that this is technically a regression from 2.6.0.

      The reason I raised this as 'Critical' is that Public Addressability with the Couchbase Operator comes in 1.2, due out in a few weeks and as of now the most used SDK won't work with this feature, which may catch some users out.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Michael Nitschinger: can we prioritize this for 1.7.5. Let me know if you see a conflict.

          ingenthr Matt Ingenthron added a comment - Michael Nitschinger : can we prioritize this for 1.7.5. Let me know if you see a conflict.

          Matt Ingenthron yes let's do that. We have a couple other things pending for 1.7.5 as well and this fits right in there I think.

          daschl Michael Nitschinger added a comment - Matt Ingenthron yes let's do that. We have a couple other things pending for 1.7.5 as well and this fits right in there I think.

          Okay, looked into the first issue and I fear there is not much we can do other than properly setting up DNS: Java, if given a hostname, will have to use DNS to look up the IP. if an IP is provided as part of the config from the cluster instead of a hostname it will not do a DNS Lookup attempt.

          So in this case either there is a way for the cluster to return as an IP instead of the hostname or DNS needs to be properly setup so the java code can resolve it.

          Now looking into the second issue.

          daschl Michael Nitschinger added a comment - Okay, looked into the first issue and I fear there is not much we can do other than properly setting up DNS: Java, if given a hostname, will have to use DNS to look up the IP. if an IP is provided as part of the config from the cluster instead of a hostname it will not do a DNS Lookup attempt. So in this case either there is a way for the cluster to return as an IP instead of the hostname or DNS needs to be properly setup so the java code can resolve it. Now looking into the second issue.

          Back to you for couple AI's (see bottom of the ticket)

          daschl Michael Nitschinger added a comment - Back to you for couple AI's (see bottom of the ticket)
          matt.carabine Matt Carabine added a comment - - edited

          Attached trace-level logging ( altaddress.log) and also re-setup cluter with the same creds (will just keep it running for now).

          Regarding your point:

          Okay, looked into the first issue and I fear there is not much we can do other than properly setting up DNS: Java, if given a hostname, will have to use DNS to look up the IP. if an IP is provided as part of the config from the cluster instead of a hostname it will not do a DNS Lookup attempt.

          The problem here is that the DNS is only resolvable from within the Kubernetes cluster, the client machines don't have access to that DNS. That's why we have to use alternate addresses to advertise a second address that IS accessible from the client, sorry if I didn't make that clear initially.
          It seems that if the original hostname (which the client can't route to even if it could resolve it) is not resolvable then it will fail, rather than appropriately falling back to the alternate addresses specified.
          Go + Libcouchbase all work fine in this setup and are able to fall back appropriately.

          matt.carabine Matt Carabine added a comment - - edited Attached trace-level logging ( altaddress.log ) and also re-setup cluter with the same creds (will just keep it running for now). Regarding your point: Okay, looked into the first issue and I fear there is not much we can do other than properly setting up DNS: Java, if given a hostname, will have to use DNS to look up the IP. if an IP is provided as part of the config from the cluster instead of a hostname it will not do a DNS Lookup attempt. The problem here is that the DNS is only resolvable from within the Kubernetes cluster, the client machines don't have access to that DNS. That's why we have to use alternate addresses to advertise a second address that IS accessible from the client, sorry if I didn't make that clear initially. It seems that if the original hostname (which the client can't route to even if it could resolve it) is not resolvable then it will fail, rather than appropriately falling back to the alternate addresses specified. Go + Libcouchbase all work fine in this setup and are able to fall back appropriately.

          People

            daschl Michael Nitschinger
            matt.carabine Matt Carabine
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty