Details
-
Bug
-
Resolution: Won't Do
-
Major
-
None
-
None
-
None
-
1
Description
While running the jepsen tests, we noticed that the SDK fails while creating connection with the cluster with the following errors.
2019-09-23 10:33:51,849{GMT} INFO [jepsen worker 9856] couchbase.cbclients: Opening new client
|
2019-09-23 10:33:52,389{GMT} INFO [cb-events] com.couchbase.core: [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.0.0-alpha.7","clientGitHash":"6caa093c","coreVersion":"2.0.0-alpha.7","coreGitHash":"6caa093c","userAgent":"couchbase-java/3.0.0-alpha.7 (Linux 3.10.0-957.1.3.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_222-b10)","seedNodes":[{"address":"172.23.104.255"}],"ioEnvironment":{"nativeIoEnabled":true,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"allowedSaslMechs":["PLAIN","CRAM_MD5","SCRAM_SHA1","SCRAM_SHA256","SCRAM_SHA512"],"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":false,"configPollIntervalMillis":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled"},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"certAuthEnabled":false,"hasTrustCertificates":false,"trustManagerFactory":null},"timeoutConfig":{"kvMs":1500,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000},"serviceConfig":{"kv":{"min":1,"max":1,"idleTimeMs":0},"query":{"min":0,"max":12,"idleTimeMs":300000},"view":{"min":0,"max":12,"idleTimeMs":300000},"search":{"min":0,"max":12,"idleTimeMs":300000},"analytics":{"min":0,"max":12,"idleTimeMs":300000}},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"credentials":"UsernameAndPassword","retryStrategy":"BestEffortRetryStrategy"} {"coreId":3}
|
2019-09-23 10:33:52,389{GMT} INFO [cb-events] com.couchbase.node: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":3,"managerPort":"8091","remote":"172.23.104.255"}
|
2019-09-23 10:33:53,014{GMT} INFO [cb-events] com.couchbase.node: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":3,"managerPort":"8091","remote":"172.23.105.197"}
|
2019-09-23 10:33:53,015{GMT} INFO [cb-events] com.couchbase.node: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":3,"managerPort":"8091","remote":"172.23.105.2"}
|
2019-09-23 10:33:53,172{GMT} INFO [cb-events] com.couchbase.node: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":3,"managerPort":"8091","remote":"172.23.105.3"}
|
2019-09-23 10:33:53,386{GMT} INFO [cb-events] com.couchbase.node: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":3,"managerPort":"8091","remote":"172.23.105.30"}
|
2019-09-23 10:33:53,491{GMT} INFO [cb-events] com.couchbase.node: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":3,"managerPort":"8091","remote":"172.23.105.41"}
|
2019-09-23 10:33:55,239{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":52,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
2019-09-23 10:33:55,449{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":54,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
2019-09-23 10:33:55,556{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":63,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
2019-09-23 10:33:55,556{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":64,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
2019-09-23 10:33:55,557{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":65,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
2019-09-23 10:33:56,448{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":71,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
2019-09-23 10:33:56,448{GMT} INFO [cb-events] com.couchbase.request: [com.couchbase.request][RequestNotRetriedEvent] Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE) {"retried":0,"reason":"NO_MORE_RETRIES (NODE_NOT_AVAILABLE)","requestId":73,"timeoutMs":2500,"service":{"type":"kv","key":""},"cancelled":true,"coreId":3,"completed":true}
|
This is intermittent and does not happen every time. But is fairly regular when running the jepsen tests with larger number of documents
Test to run (repeatedly running the test will cause the issue to reproduce):
lein trampoline run test --nodes-file ./nodes --username root --password couchbase --workload=rebalance --scenario=sequential-rebalance-out-in --node-count=6 --no-autofailover --replicas=1 --disrupt-count=1 --kv-timeout=1.5 --durability=0:100:0:0 --doc-count=5000 --doc-threads=1