Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34762

[System Test] CBAS crash due to java.lang.ArrayIndexOutOfBoundsException: Index 1704027 out of bounds for length 32768

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-3614
      Test : -test tests/analytics/test_analytics_rebalance.yml -scope tests/analytics/scope_analytics_rebalance.yml
      Scale : 4
      Iteration : 1

      cbas service crashed on 172.23.104.21 due to java.lang.ArrayIndexOutOfBoundsException: Index 1704027 out of bounds for length 32768 error.

      2019-06-26T11:50:03.411-07:00 ERRO CBAS.tcp.TCPEndpoint [TCPEndpoint IO Thread [/0.0.0.0:9116]] Unexpected tcp io error in connection TCPConnection[Remote Address: /172.23.104.94:9116 Local Address: /0.0.0.0:9116]
      		 
      java.io.IOException: Connection reset by peer
      		 
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
      		 
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
      		 
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
      		 
              at sun.nio.ch.IOUtil.read(IOUtil.java:245) ~[?:?]
      		 
              at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
      		 
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
      		 
              at org.apache.hyracks.ipc.sockets.PlainSocketChannel.read(PlainSocketChannel.java:47) ~[hyracks-ipc.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.muxdemux.FullFrameChannelReadInterface.read(FullFrameChannelReadInterface.java:79) ~[hyracks-net.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.read(ChannelControlBlock.java:95) ~[hyracks-net.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.driveReaderStateMachine(MultiplexedConnection.java:417) ~[hyracks-net.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.notifyIOReady(MultiplexedConnection.java:119) ~[hyracks-net.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:197) [hyracks-net.jar:6.5.0-3614]
      		 
      2019-06-26T11:50:03.428-07:00 ERRO CBAS.tcp.TCPEndpoint [TCPEndpoint IO Thread [/0.0.0.0:9116]] Unexpected tcp io error in connection TCPConnection[Remote Address: /172.23.104.94:50632 Local Address: /0.0.0.0:9116]
      		 
      java.io.IOException: Connection reset by peer
      		 
              at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
      		 
              at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
      		 
              at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
      		 
              at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
      		 
              at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
      		 
              at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
      		 
              at org.apache.hyracks.ipc.sockets.PlainSocketChannel.read(PlainSocketChannel.java:47) ~[hyracks-ipc.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.driveReaderStateMachine(MultiplexedConnection.java:358) ~[hyracks-net.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.notifyIOReady(MultiplexedConnection.java:119) ~[hyracks-net.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:197) [hyracks-net.jar:6.5.0-3614]
      		 
      2019-06-26T11:50:03.432-07:00 FATA CBAS.runtime.DcpUpdateCallback [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.2959:TAID:TID:ANID:ODID:2:0:1:0:0] Restarting process to ensure data integrity
      		 
      org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.ArrayIndexOutOfBoundsException: Index 1704027 out of bounds for length 32768
      		 
              at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:51) ~[hyracks-api.jar:6.5.0-3614]
      		 
              at org.apache.asterix.runtime.operators.LSMPrimaryUpsertOperatorNodePushable$1.process(LSMPrimaryUpsertOperatorNodePushable.java:196) [asterix-runtime.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.processFrame(LSMHarness.java:854) ~[hyracks-storage-am-lsm-common.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.batchOperate(LSMHarness.java:705) [hyracks-storage-am-lsm-common.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.batchOperate(LSMTreeIndexAccessor.java:214) [hyracks-storage-am-lsm-common.jar:6.5.0-3614]
      		 
              at org.apache.asterix.runtime.operators.LSMPrimaryUpsertOperatorNodePushable.nextFrame(LSMPrimaryUpsertOperatorNodePushable.java:323) [asterix-runtime.jar:6.5.0-3614]
      		 
              at org.apache.asterix.external.feed.dataflow.SyncFeedRuntimeInputHandler.nextFrame(SyncFeedRuntimeInputHandler.java:46) [asterix-external-data.jar:6.5.0-3614]
      		 
              at org.apache.asterix.external.operators.FeedMetaStoreNodePushable.nextFrame(FeedMetaStoreNodePushable.java:151) [asterix-external-data.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:401) [hyracks-control-nc.jar:6.5.0-3614]
      		 
              at org.apache.hyracks.control.nc.Task.run(Task.java:335) [hyracks-control-nc.jar:6.5.0-3614]
      		 
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      		 
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      		 
              at java.lang.Thread.run(Thread.java:834) [?:?]
      		 
      Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1704027 out of bounds for length 32768
      		 
              at org.apache.hyracks.storage.am.bloomfilter.impls.MurmurHash128Bit.hash3_x64_128(MurmurHash128Bit.java:68) ~[hyracks-storage-am-bloomfilter.jar:6.5.0-3614]
      		 
              at org.apache.asterix.common.transactions.AbstractOperationCallback.computePrimaryKeyHashValue(AbstractOperationCallback.java:47) ~[asterix-common.jar:6.5.0-3614]
      		 
              at org.apache.asterix.transaction.management.opcallbacks.LockThenSearchOperationCallback.before(LockThenSearchOperationCallback.java:67) ~[asterix-transactions.jar:6.5.0-3614]
      		 
              at org.apache.asterix.runtime.operators.LSMPrimaryUpsertOperatorNodePushable$1.process(LSMPrimaryUpsertOperatorNodePushable.java:175) ~[asterix-runtime.jar:6.5.0-3614]
      		 
              ... 11 more
      		 
      2019-06-26T11:50:03.999-07:00 FATA CBAS.util.ExitUtil [org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.2959:TAID:TID:ANID:ODID:2:0:1:0:0] JVM halting with status 103; thread dump at halt:
      		 
      "main" [tid=1 state=WAITING lock=java.util.concurrent.Semaphore$NonfairSync@79e17104]

      Attachments

        Issue Links

          links to

          Web Link *DB review

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty