Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Columnar 1.0.0
-
1.0.0-2085 4-node cluster ( 8 VCPUs / 32 G instances)
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 43
Description
Errors as seen on node 003 -
2024-05-21T20:57:28.740+00:00 ERRO CBAS.DefaultPromise.rejectedExecution [cb-io-kv-61-2] Failed to submit a listener notification task. Event loop shut down? |
java.util.concurrent.RejectedExecutionException: event executor terminated
|
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:862) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:500) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.endpoint.BaseEndpoint.channelFutureIntoMono(BaseEndpoint.java:700) ~[core-io-2.5.2.jar:?] |
at com.couchbase.client.core.endpoint.BaseEndpoint.lambda$reconnect$1(BaseEndpoint.java:341) ~[core-io-2.5.2.jar:?] |
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:45) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:77) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.Mono.subscribe(Mono.java:4480) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.Mono.subscribeWith(Mono.java:4561) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.Mono.subscribe(Mono.java:4462) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.Mono.subscribe(Mono.java:4398) [reactor-core-3.5.8.jar:3.5.8] |
at reactor.core.publisher.Mono.subscribe(Mono.java:4370) [reactor-core-3.5.8.jar:3.5.8] |
at com.couchbase.client.core.endpoint.BaseEndpoint.reconnect(BaseEndpoint.java:394) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.endpoint.BaseEndpoint.connect(BaseEndpoint.java:261) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.endpoint.BaseEndpoint.notifyChannelInactive(BaseEndpoint.java:537) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.io.netty.PipelineErrorHandler.channelInactive(PipelineErrorHandler.java:66) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.io.netty.kv.KeyValueMessageHandler.channelInactive(KeyValueMessageHandler.java:294) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1174) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasksFrom(SingleThreadEventExecutor.java:426) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:375) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:763) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:431) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [core-io-2.5.2.jar:?] |
at com.couchbase.client.core.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [core-io-2.5.2.jar:?] |
at java.base/java.lang.Thread.run(Thread.java:840) [?:?] |
A bit of background - Copy to KV command was attempted 3 times (firstly at 024-05-21T19:34:13.507Z, secondly at 2024-05-21T20:47:59.345Z and finally at 2024-05-21T22:04:07.286Z). During one of these operations, the bucket that was being used for copy to KV ran out of memory (Temp OOM errors). Not really sure if the error message has anything to do with this.
There appears to be another deadlock error a little later( 22.07). I'll log a separate ticket for that. If the RCA determines that the issue is related to this , please close it as a duplicate.
Logs ->
https://cb-engineering.s3.amazonaws.com/SysTestColumnar21May/collectinfo-2024-05-22T072859-ns_1%40svc-da-node-001.mmbvoelwhg-otdos.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar21May/collectinfo-2024-05-22T072859-ns_1%40svc-da-node-002.mmbvoelwhg-otdos.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar21May/collectinfo-2024-05-22T072859-ns_1%40svc-da-node-003.mmbvoelwhg-otdos.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar21May/collectinfo-2024-05-22T072859-ns_1%40svc-da-node-004.mmbvoelwhg-otdos.sandbox.nonprod-project-avengers.com.zip
Attachments
Issue Links
- is caused by
-
-
- Closed
-