Details
Major Description
Received the following error Exception (only registered as a Warning perhaps?):
WARNING: [/192.168.27.103:11210][KeyValueEndpoint]: Caught unknown exception: java.lang.ArrayIndexOutOfBoundsException: 29289
com.couchbase.client.deps.io.netty.handler.codec.DecoderException: java.lang.ArrayIndexOutOfBoundsException: 29289
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
I have a sample application which reproduces the problem during a rebalance.
What the app does:
- Creates an ArrayList of sample documents to work with
- Loop pushing an async upsert to Couchbase once every ~100ms.
- Upsert uses ReplicateTo.ONE
- If it fails, a retry will be triggered with a backoff mechanism (kind of linear-ish, that’s on my to do list to fix).
- CB_HOST variable contains the machine to connect to (note: that is not the machine I have been adding/removing from cluster in rebalance tests).
Code is available on github.
I’ve done the dirty/lazy thing of having all the .jar dependencies in the repo as it was easier for me pushing it around between machines.
Might take you a bit longer to clone because of that though, sorry!
https://github.com/tom-cb/cb-java2.git
Compile:
$ javac -cp apache-commons/commons-math3-3.3/commons-math3-3.3.jar:Couchbase-Java-Client-2.0.1/couchbase-java-client-2.0.1.jar:Couchbase-Java-Client-2.0.1/couchbase-core-io-1.0.1.jar:Couchbase-Java-Client-2.0.1/rxjava-1.0.0-rc.9.jar RetryWithDelay.java HelloCouchbase.java 2>&1
Run:
$ java -cp apache-commons/commons-math3-3.3/commons-math3-3.3.jar:Couchbase-Java-Client-2.0.1/couchbase-java-client-2.0.1.jar:Couchbase-Java-Client-2.0.1/couchbase-core-io-1.0.1.jar:Couchbase-Java-Client-2.0.1/rxjava-1.0.0-rc.9.jar:. HelloCouchbase 2>trace.err
Simply run the app, then trigger a rebalance whilst operations are ongoing.
I've also attached a client side log captured during a run which triggers the exception.
Excerpt:
Dec 01, 2014 12:42:28 PM com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline$TailContext exceptionCaught
WARNING: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not han
dle the exception.
com.couchbase.client.deps.io.netty.handler.codec.DecoderException: java.lang.ArrayIndexOutOfBoundsException: 29289
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 29289
at com.couchbase.client.deps.io.netty.buffer.UnpooledHeapByteBuf._getByte(UnpooledHeapByteBuf.java:292)
at com.couchbase.client.deps.io.netty.buffer.UnpooledHeapByteBuf.getByte(UnpooledHeapByteBuf.java:287)
at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:428)
at com.couchbase.client.core.endpoint.kv.KeyValueHandler.decodeResponse(KeyValueHandler.java:79)
at com.couchbase.client.core.endpoint.AbstractGenericHandler.decode(AbstractGenericHandler.java:152)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageCodec$2.decode(MessageToMessageCodec.java:81)
at com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89)
... 19 more