Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 6.0.0
Affects Version/s: 6.0.0
Component/s: analytics
Labels:
- beta
- triaged

Triage:
Untriaged
Is this a Regression?:
Unknown
Sprint:
CX Sprint 115

Description

Deadlock threads:

"nioEventLoopGroup-3-7" [tid=94 state=BLOCKED lock=com.couchbase.client.dcp.events.DefaultEventBus@2678e76c lockOwner="nioEventLoopGroup-3-3" (tid=90]

	at com.couchbase.client.dcp.events.DefaultEventBus.publish(DefaultEventBus.java:13)

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.updateFailoverLog(DcpChannelControlMessageHandler.java:141)

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.handleOpenStreamResponse(DcpChannelControlMessageHandler.java:90)

	- <locked com.couchbase.client.dcp.conductor.DcpChannel@5d208841>

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.onEvent(DcpChannelControlMessageHandler.java:39)

	at com.couchbase.client.dcp.transport.netty.DcpMessageHandler.channelRead(DcpMessageHandler.java:120)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:299)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:415)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)

	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)

	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

	at java.lang.Thread.run(Thread.java:748)

"nioEventLoopGroup-3-7" [tid=94 state=BLOCKED lock=com.couchbase.client.dcp.events.DefaultEventBus@2678e76c lockOwner="nioEventLoopGroup-3-3" (tid=90]

	at com.couchbase.client.dcp.events.DefaultEventBus.publish(DefaultEventBus.java:13)

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.updateFailoverLog(DcpChannelControlMessageHandler.java:141)

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.handleOpenStreamResponse(DcpChannelControlMessageHandler.java:90)

	- <locked com.couchbase.client.dcp.conductor.DcpChannel@5d208841>

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.onEvent(DcpChannelControlMessageHandler.java:39)

	at com.couchbase.client.dcp.transport.netty.DcpMessageHandler.channelRead(DcpMessageHandler.java:120)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:299)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:415)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)

	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)

	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

	at java.lang.Thread.run(Thread.java:748)

"nioEventLoopGroup-3-3" [tid=90 state=BLOCKED lock=java.util.concurrent.ConcurrentHashMap@588e657d lockOwner="Executor-10:5519bb39d1d16623dbef730b70c94737:JID:0.10:TAID:TID:ANID:ODID:0:0:0:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor" (tid=70]

	at com.couchbase.client.dcp.conductor.Conductor.startStreamForPartition(Conductor.java:147)

	at com.couchbase.client.dcp.Client.startStreaming(Client.java:272)

	at com.couchbase.analytics.adapter.CouchbaseConnector.handleEvent(CouchbaseConnector.java:153)

	at com.couchbase.analytics.adapter.CouchbaseConnector$$Lambda$320/1088448545.onEvent(Unknown Source)

	at com.couchbase.client.dcp.events.DefaultEventBus.publish(DefaultEventBus.java:14)

	- <locked com.couchbase.client.dcp.events.DefaultEventBus@2678e76c>

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.handleDcpStreamEndMessage(DcpChannelControlMessageHandler.java:174)

	at com.couchbase.client.dcp.conductor.DcpChannelControlMessageHandler.onEvent(DcpChannelControlMessageHandler.java:43)

	at com.couchbase.client.dcp.transport.netty.DcpMessageHandler.channelRead(DcpMessageHandler.java:120)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:299)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:415)

	at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)

	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

	at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)

	at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)

	at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)

	at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)

	at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)

	at com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

	at java.lang.Thread.run(Thread.java:748)

"FixerThread:cbas:beer-sample:5519bb39d1d16623dbef730b70c94737:0" [tid=105 state=BLOCKED lock=java.util.concurrent.ConcurrentHashMap@588e657d lockOwner="Executor-10:5519bb39d1d16623dbef730b70c94737:JID:0.10:TAID:TID:ANID:ODID:0:0:0:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor" (tid=70]

	at com.couchbase.client.dcp.conductor.Conductor.reviveDeadConnections(Conductor.java:277)

	at com.couchbase.client.dcp.events.DeadConnectionDetection.run(DeadConnectionDetection.java:36)

	at com.couchbase.client.dcp.conductor.Fixer.run(Fixer.java:75)

	at java.lang.Thread.run(Thread.java:748)

Logs are attached. I had a debugger with a break point on stats related section, not sure if that caused it.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

collectinfo-2018-08-08T170753-n_0@127.0.0.1.zip
8.85 MB
08/Aug/18 10:40 AM
collectinfo-2018-08-08T170753-n_1@127.0.0.1.zip
6.43 MB
08/Aug/18 10:40 AM
collectinfo-2018-08-08T170753-n_2@127.0.0.1.zip
6.57 MB
08/Aug/18 10:40 AM

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

For Gerrit Dashboard: MB-30819
#	Subject	Branch	Project	Status	CR	V
98023,12	MB-30819 Fix deadlock when reopening a stream	master	asterix-opt	Status: MERGED	+2	+1

Activity

People

Assignee:: Abdullah Alamoudi [X] (Inactive)

Reporter:: Murtadha Hubail

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 08/Aug/18 10:43 AM

Updated:: 15/Aug/18 6:45 PM

Resolved:: 10/Aug/18 4:04 PM

Gerrit Reviews

There are no open Gerrit changes

Show There is 1 closed Gerrit change

Hide There is 1 closed Gerrit change

MB-30819 Fix deadlock when reopening a stream: Gerrit Review:

[CX] Possible Deadlock in DefaultEventBus/DcpChannel

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty