Major Description
CouchbaseConnectorStressTest.testConnectDisconnectCycle performs the following in a loop:
Client client = Client.configure()
|
.eventLoopGroup(ioPool())
|
.bucket(bucket)
|
.password(cbPassword)
|
.hostnames(cbNodes)
|
.connectionNameGenerator(() -> threadName)
|
.controlParam(DcpControl.Names.CONNECTION_BUFFER_SIZE, 16384)
|
.bufferAckWatermark(80)
|
.build();
|
|
|
client.controlEventHandler(ReferenceCountUtil::release);
|
client.dataEventHandler(event -> {
|
try {
|
client.acknowledgeBuffer(event);
|
} finally {
|
ReferenceCountUtil.release(event);
|
}
|
});
|
|
|
timedAwait("client.connect()", client.connect());
|
timedAwait("client.disconnect()", client.disconnect());
|
Between 10-20 seconds of iterations, I observe many exceptions like the following on the console:
## testConnectDisconnectCycle START
|
### Starting 4 threads with run time of 60 seconds.
|
#### [thread 0] Starting thread (will trace every 10 seconds)
|
#### [thread 1] Starting thread (will trace every 10 seconds)
|
#### [thread 2] Starting thread (will trace every 10 seconds)
|
#### [thread 3] Starting thread (will trace every 10 seconds)
|
#### [thread 1] ...iteration 1338...
|
#### [thread 0] ...iteration 1001...
|
#### [thread 3] ...iteration 213...
|
#### [thread 2] ...iteration 450...
|
[cb-io-1-6] 19:58:26 WARN DefaultChannelPipeline:299 - An exception 'java.lang.IllegalStateException: complete already: DefaultChannelPromise@752e0392(failure: java.io.IOException: Connection reset by peer)' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
|
java.io.IOException: Connection reset by peer
|
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
|
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
|
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
|
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
|
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
|
at com.couchbase.client.deps.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
|
at com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
|
at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
|
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399)
|
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
|
at java.lang.Thread.run(Thread.java:745)
|
The error does not get thrown to the client, and iterations continue. Eventually after 30-40 seconds (and 1000s of iterations), I do get an error out of all threads:
#### [thread 3] ...iteration 1801...
|
#### [thread 2] Failing fast due to 1 failure(s): [com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket]
|
#### [thread 2] Exiting thread after 2320 iterations.
|
#### [thread 1] Failing fast due to 1 failure(s): [com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket]
|
#### [thread 1] Exiting thread after 2872 iterations.
|
#### [thread 3] Failing fast due to 1 failure(s): [com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket]
|
#### [thread 3] Exiting thread after 1808 iterations.
|
### Completed 8792 iterations with 4 failure(s).
|
## testConnectDisconnectCycle FAILED (com.couchbase.client.dcp.error.BootstrapException)
|
|
|
com.couchbase.client.dcp.error.BootstrapException: Could not connect to Cluster/Bucket
|
|
|
at com.couchbase.client.dcp.Client$4.call(Client.java:290)
|
at com.couchbase.client.dcp.Client$4.call(Client.java:287)
|
at rx.Completable$24$1.onError(Completable.java:1735)
|
at rx.internal.operators.CompletableOnSubscribeConcatArray$ConcatInnerSubscriber.onError(CompletableOnSubscribeConcatArray.java:63)
|
at com.couchbase.client.dcp.conductor.HttpStreamingConfigProvider$4$1.operationComplete(HttpStreamingConfigProvider.java:194)
|
at com.couchbase.client.dcp.conductor.HttpStreamingConfigProvider$4$1.operationComplete(HttpStreamingConfigProvider.java:177)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:507)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:486)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:427)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:120)
|
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
|
at com.couchbase.client.dcp.transport.netty.ConnectInterceptingHandler.exceptionCaught(ConnectInterceptingHandler.java:108)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:290)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:269)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:261)
|
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:423)
|
at com.couchbase.client.deps.io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
|
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:144)
|
at com.couchbase.client.deps.io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
|
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:230)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:290)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:269)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:261)
|
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1261)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:290)
|
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:269)
|
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:899)
|
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
|
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:572)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:513)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:427)
|
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:399)
|
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
|
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
|
at java.lang.Thread.run(Thread.java:745)
|
Caused by: java.io.IOException: Connection reset by peer
|
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
|
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
|
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
|
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
|
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
|
at com.couchbase.client.deps.io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
|
at com.couchbase.client.deps.io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
|
at com.couchbase.client.deps.io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:241)
|
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
|
... 7 more
|