Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30819

[CX] Possible Deadlock in DefaultEventBus/DcpChannel

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.0.0
    • 6.0.0
    • analytics
    • Untriaged
    • Unknown
    • CX Sprint 115

    Description

      Deadlock threads:

       

      "nioEventLoopGroup-3-7" [tid=94 state=BLOCKED lock=com.couchbase.client.dcp.events.DefaultEventBus@2678e76c lockOwner="nioEventLoopGroup-3-3" (tid=90]
      		 
      	at com.couchbase.client.dcp.events.DefaultEventBus.publish(DefaultEventBus.java:13)
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.updateFailoverLog(DcpChannelControlMessageHandler.java:141)
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.handleOpenStreamResponse(DcpChannelControlMessageHandler.java:90)
      		 
      	- <locked com.couchbase.client.dcp.conductor.DcpChannel@5d208841>
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.onEvent(DcpChannelControlMessageHandler.java:39)
      		 
      	at com.couchbase.client.dcp.transport.netty.DcpMessageHandler.channelRead(DcpMessageHandler.java:120)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:299)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:415)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      		 
      	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
      		 
      	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
      		 
      	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      		 
      	at java.lang.Thread.run(Thread.java:748)
      		 
       
      		 
      "nioEventLoopGroup-3-7" [tid=94 state=BLOCKED lock=com.couchbase.client.dcp.events.DefaultEventBus@2678e76c lockOwner="nioEventLoopGroup-3-3" (tid=90]
      		 
      	at com.couchbase.client.dcp.events.DefaultEventBus.publish(DefaultEventBus.java:13)
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.updateFailoverLog(DcpChannelControlMessageHandler.java:141)
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.handleOpenStreamResponse(DcpChannelControlMessageHandler.java:90)
      		 
      	- <locked com.couchbase.client.dcp.conductor.DcpChannel@5d208841>
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.onEvent(DcpChannelControlMessageHandler.java:39)
      		 
      	at com.couchbase.client.dcp.transport.netty.DcpMessageHandler.channelRead(DcpMessageHandler.java:120)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:299)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:415)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      		 
      	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
      		 
      	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
      		 
      	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      		 
      	at java.lang.Thread.run(Thread.java:748)
      		 
       
      		 
       
      		 
      "nioEventLoopGroup-3-3" [tid=90 state=BLOCKED lock=java.util.concurrent.ConcurrentHashMap@588e657d lockOwner="Executor-10:5519bb39d1d16623dbef730b70c94737:JID:0.10:TAID:TID:ANID:ODID:0:0:0:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor" (tid=70]
      		 
      	at com.couchbase.client.dcp.conductor.Conductor.startStreamForPartition(Conductor.java:147)
      		 
      	at com.couchbase.client.dcp.Client.startStreaming(Client.java:272)
      		 
      	at com.couchbase.analytics.adapter.CouchbaseConnector.handleEvent(CouchbaseConnector.java:153)
      		 
      	at com.couchbase.analytics.adapter.CouchbaseConnector$$Lambda$320/1088448545.onEvent(Unknown Source)
      		 
      	at com.couchbase.client.dcp.events.DefaultEventBus.publish(DefaultEventBus.java:14)
      		 
      	- <locked com.couchbase.client.dcp.events.DefaultEventBus@2678e76c>
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.handleDcpStreamEndMessage(DcpChannelControlMessageHandler.java:174)
      		 
      	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.onEvent(DcpChannelControlMessageHandler.java:43)
      		 
      	at com.couchbase.client.dcp.transport.netty.DcpMessageHandler.channelRead(DcpMessageHandler.java:120)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:299)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:415)
      		 
      	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      		 
      	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      		 
      	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      		 
      	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
      		 
      	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
      		 
      	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
      		 
      	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      		 
      	at java.lang.Thread.run(Thread.java:748)
      		 
       
      		 
      "FixerThread:cbas:beer-sample:5519bb39d1d16623dbef730b70c94737:0" [tid=105 state=BLOCKED lock=java.util.concurrent.ConcurrentHashMap@588e657d lockOwner="Executor-10:5519bb39d1d16623dbef730b70c94737:JID:0.10:TAID:TID:ANID:ODID:0:0:0:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor" (tid=70]
      		 
      	at com.couchbase.client.dcp.conductor.Conductor.reviveDeadConnections(Conductor.java:277)
      		 
      	at com.couchbase.client.dcp.events.DeadConnectionDetection.run(DeadConnectionDetection.java:36)
      		 
      	at com.couchbase.client.dcp.conductor.Fixer.run(Fixer.java:75)
      		 
      	at java.lang.Thread.run(Thread.java:748)

      Logs are attached. I had a debugger with a break point on stats related section, not sure if that caused it.

      Attachments

        Activity

          People

            Abdullah.Alamoudi Abdullah Alamoudi [X] (Inactive)
            murtadha.hubail Murtadha Hubail
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty