Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62315

[System Test] Analytics driver exit with status 88 - MERGE operation failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • Columnar 1.0.0
    • Columnar 1.0.0
    • analytics
    • Columnar Edition 1.0.0 build 2144 4-node cluster ( 8vcpus+32 GB)
    • Untriaged
    • 0
    • Unknown
    • Analytics Sprint 44

    Description

      Seen on 003 as well as on 004-

      Logs from 003 -

      2024-06-13T13:01:20.306+00:00 ERRO CBAS.buffercache.BufferCache [Executor-394:81de45078a3bbdffa0be812d4898cb9b] Error while reading a page CachedPage:[page:3, compressedPageOffset:177438, compressedSize:86956] in file /var/cb-cache/@analytics/v_iodevice_6/storage/partition_6/Database7LWSXKGOa/scope1vGbWyAPk/remotedatasetXktDjmVy/0/remotedatasetXktDjmVy/0_44_b
      2024-06-13T13:01:20.306+00:00 WARN CBAS.buffercache.BufferCache [Executor-394:81de45078a3bbdffa0be812d4898cb9b] Failure while trying to read a page from disk
      org.apache.hyracks.api.exceptions.HyracksDataException: java.net.SocketException: Connection reset
      	at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.column.cloud.buffercache.read.CloudMegaPageReadContext.readFromStream(CloudMegaPageReadContext.java:172) ~[hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.column.cloud.buffercache.read.CloudMegaPageReadContext.processHeader(CloudMegaPageReadContext.java:124) ~[hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.file.CompressedBufferedFileHandle.read(CompressedBufferedFileHandle.java:62) ~[hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:571) ~[hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.buffercache.BufferCache.tryRead(BufferCache.java:543) ~[hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:213) [hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.column.cloud.buffercache.read.CloudColumnReadContext.pin(CloudColumnReadContext.java:205) [hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.column.cloud.buffercache.read.CloudColumnReadContext.pinAll(CloudColumnReadContext.java:150) [hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.column.cloud.buffercache.read.CloudColumnReadContext.prepareColumns(CloudColumnReadContext.java:141) [hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.column.impls.btree.ColumnBTreeRangeSearchCursor.doOpen(ColumnBTreeRangeSearchCursor.java:134) [hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.EnforcedIndexCursor.open(EnforcedIndexCursor.java:54) [hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.btree.impls.DiskBTree.searchDown(DiskBTree.java:138) [hyracks-storage-am-btree.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.btree.impls.DiskBTree.search(DiskBTree.java:107) [hyracks-storage-am-btree.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.btree.impls.DiskBTree$DiskBTreeAccessor.search(DiskBTree.java:195) [hyracks-storage-am-btree.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.util.IndexCursorUtils.open(IndexCursorUtils.java:90) [hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.doOpen(LSMBTreeRangeSearchCursor.java:415) [hyracks-storage-am-lsm-btree.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.common.EnforcedIndexCursor.open(EnforcedIndexCursor.java:54) [hyracks-storage-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.search(LSMBTree.java:219) [hyracks-storage-am-lsm-btree.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.doMerge(LSMBTree.java:321) [hyracks-storage-am-lsm-btree.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:917) [hyracks-storage-am-lsm-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.doIo(LSMHarness.java:566) [hyracks-storage-am-lsm-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:608) [hyracks-storage-am-lsm-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.merge(LSMTreeIndexAccessor.java:128) [hyracks-storage-am-lsm-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:52) [hyracks-storage-am-lsm-common.jar:1.0.0-2144]
      	at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:33) [hyracks-storage-am-lsm-common.jar:1.0.0-2144]
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
      	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
      Caused by: java.net.SocketException: Connection reset
      	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328) ~[?:?]
      	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) ~[?:?]
      	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808) ~[?:?]
      	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketInputRecord.readFully(SSLSocketInputRecord.java:467) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:243) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181) ~[?:?]
      	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1513) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1484) ~[?:?]
      	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069) ~[?:?]
      	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.16.jar:4.4.16]
      	at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:197) ~[httpcore-4.4.16.jar:4.4.16]
      	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) ~[httpcore-4.4.16.jar:4.4.16]
      	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) ~[httpclient-4.5.14.jar:4.5.14]
      	at java.base/java.io.FilterInputStream.read(FilterInputStream.java:132) ~[?:?]
      	at software.amazon.awssdk.services.s3.internal.checksums.S3ChecksumValidatingInputStream.read(S3ChecksumValidatingInputStream.java:112) ~[s3-2.24.9.jar:?]
      	at java.base/java.io.FilterInputStream.read(FilterInputStream.java:132) ~[?:?]
      	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66) ~[sdk-core-2.24.9.jar:?]
      	at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingInputStream.read(BytesReadTrackingInputStream.java:49) ~[sdk-core-2.24.9.jar:?]
      	at java.base/java.io.FilterInputStream.read(FilterInputStream.java:132) ~[?:?]
      	at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:66) ~[sdk-core-2.24.9.jar:?]
      	at org.apache.hyracks.storage.am.lsm.btree.column.cloud.buffercache.read.CloudMegaPageReadContext.readFromStream(CloudMegaPageReadContext.java:165) ~[hyracks-storage-am-lsm-btree-column.jar:1.0.0-2144]
      	... 28 more
      2024-06-13T13:01:20.307+00:00 ERRO CBAS.read.CloudColumnReadContext [Executor-394:81de45078a3bbdffa0be812d4898cb9b] Error while pinning page number 3 with number of pages 20. 
      

      cbcollect ->

      https://cb-engineering.s3.amazonaws.com/SysTestColumnar13Jun/collectinfo-2024-06-13T162907-ns_1%40svc-da-node-001.smlgzhhxmxj2aadc.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar13Jun/collectinfo-2024-06-13T162907-ns_1%40svc-da-node-002.smlgzhhxmxj2aadc.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar13Jun/collectinfo-2024-06-13T162907-ns_1%40svc-da-node-003.smlgzhhxmxj2aadc.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar13Jun/collectinfo-2024-06-13T162907-ns_1%40svc-da-node-004.smlgzhhxmxj2aadc.sandbox.nonprod-project-avengers.com.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              wail.alkowaileet Wail Alkowaileet (Inactive)
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty