Details
-
Bug
-
Resolution: Fixed
-
Critical
-
2.2.4
-
None
-
None
-
1
Description
Issue
Many management tests (Bucket, User, Collection i.e. http based) are consistently failing on Jenkins (and locally) since the rate limiting/quota patch was merged (JVMCBC-1032), with a CouchbaseException with message: "Unknown manager error: ...". The Jenkins report show all the current errors.
Example
Here is a Bucket manager error for the test createAndDropBucket:
com.couchbase.client.core.error.CouchbaseException: Unknown manager error: Requested resource not found.
|
{"completed":true,"coreId":"0xa8dfe36c00000001","httpBody":"Requested resource not found.\r\n","httpStatus":404,"idempotent":true,"lastDispatchedFrom":"192.168.100.174:60742","lastDispatchedTo":"172.23.111.136:8091","requestId":228,"requestType":"CoreHttpRequest","retried":0,"service":{"method":"GET","path":"/pools/default/buckets/4d7b3f08-b56e-4e25-93aa-2578d10e4755","type":"mgmt"},"status":"NOT_FOUND","timeoutMs":75000,"timings":{"dispatchMicros":150878,"totalDispatchMicros":150878,"totalMicros":151281}} |
at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51) |
at com.couchbase.client.java.manager.bucket.BucketManager.getBucket(BucketManager.java:70) |
at com.couchbase.client.java.manager.bucket.BucketManagerIntegrationTest.lambda$waitUntilDropped$1(BucketManagerIntegrationTest.java:112) |
at org.awaitility.core.CallableCondition$ConditionEvaluationWrapper.eval(CallableCondition.java:99) |
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:234) |
at org.awaitility.core.ConditionAwaiter$ConditionPoller.call(ConditionAwaiter.java:221) |
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) |
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) |
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) |
at java.base/java.lang.Thread.run(Thread.java:834) |
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
|
at com.couchbase.client.core.io.netty.manager.NonChunkedManagerMessageHandler.failRequestWith(NonChunkedManagerMessageHandler.java:55) |
at com.couchbase.client.core.io.netty.NonChunkedHttpMessageHandler.channelRead(NonChunkedHttpMessageHandler.java:240) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) |
at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.io.netty.TrafficCaptureHandler.channelRead(TrafficCaptureHandler.java:64) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) |
at com.couchbase.client.core.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) |
at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) |
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) |
... 1 more |
Suppressed: com.couchbase.client.core.error.CouchbaseException: Unknown manager error: Requested resource not found.
|
{"completed":true,"coreId":"0xa8dfe36c00000001","httpBody":"Requested resource not found.\r\n","httpStatus":404,"idempotent":false,"lastDispatchedFrom":"192.168.100.174:60740","lastDispatchedTo":"172.23.111.138:8091","requestId":229,"requestType":"CoreHttpRequest","retried":0,"service":{"method":"DELETE","path":"/pools/default/buckets/4d7b3f08-b56e-4e25-93aa-2578d10e4755","type":"mgmt"},"status":"NOT_FOUND","timeoutMs":75000,"timings":{"dispatchMicros":149088,"totalDispatchMicros":149088,"totalMicros":149267}} |
at com.couchbase.client.java.AsyncUtils.block(AsyncUtils.java:51) |
at com.couchbase.client.java.manager.bucket.BucketManager.dropBucket(BucketManager.java:62) |
at com.couchbase.client.java.manager.bucket.BucketManagerIntegrationTest.dropBuckets(BucketManagerIntegrationTest.java:82) |
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) |
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |
at java.base/java.lang.reflect.Method.invoke(Method.java:566) |
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688) |
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) |
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) |
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) |
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126) |
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptAfterEachMethod(TimeoutExtension.java:108) |
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) |
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) |
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) |
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) |
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) |
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) |
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) |
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) |
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeMethodInExtensionContext(ClassBasedTestDescriptor.java:490) |
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$synthesizeAfterEachMethodAdapter$20(ClassBasedTestDescriptor.java:480) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAfterEachMethods$9(TestMethodTestDescriptor.java:236) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$12(TestMethodTestDescriptor.java:269) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeAllAfterMethodsOrCallbacks$13(TestMethodTestDescriptor.java:269) |
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAllAfterMethodsOrCallbacks(TestMethodTestDescriptor.java:268) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeAfterEachMethods(TestMethodTestDescriptor.java:234) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135) |
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129) |
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84) |
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) |
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129) |
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84) |
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) |
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129) |
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127) |
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126) |
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84) |
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32) |
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) |
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51) |
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108) |
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88) |
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54) |
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67) |
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52) |
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96) |
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75) |
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69) |
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) |
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) |
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) |
Suppressed: java.lang.Exception: The above exception was originally thrown by another thread at the following location.
|
at com.couchbase.client.core.io.netty.manager.NonChunkedManagerMessageHandler.failRequestWith(NonChunkedManagerMessageHandler.java:55) |
at com.couchbase.client.core.io.netty.NonChunkedHttpMessageHandler.channelRead(NonChunkedHttpMessageHandler.java:240) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) |
at com.couchbase.client.core.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.io.netty.TrafficCaptureHandler.channelRead(TrafficCaptureHandler.java:64) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) |
at com.couchbase.client.core.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) |
at com.couchbase.client.core.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) |
at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) |
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) |
at java.base/java.lang.Thread.run(Thread.java:834) |
|
In all of these new test failures, the request is retried 0 times, probably due to not having the expected exception type.
Environment
The failures occur against all versions of the CB Server (7.1 - 5.5).