Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62715

Failed to READ 92204 bytes from stream remote_p0yMY_volCollection_4_dzkqi/0_418_b, streamOffset: 1522705899, streamRemaining: 196526. Multiple JVM halts with 103.

    XMLWordPrintable

Details

    Description

      [root@ip-10-3-6-27 logs]# grep WARN analytics_info.log | grep streamOffset
      		 
      2024-07-12T17:01:01.949+00:00 WARN CBAS.read.CloudMegaPageReadContext [SA:JID:1.434:TAID:TID:ANID:ODID:1:0:26:0:0] Failed to READ 92204 bytes from stream {file: /var/cb-cache/@analytics/v_iodevice_10/storage/partition_26/Default/Default/remote_p0yMY_volCollection_4_dzkqi/0/remote_p0yMY_volCollection_4_dzkqi/0_418_b, streamOffset: 1522705899, streamRemaining: 196526}
      		 
      2024-07-12T17:01:01.951+00:00 WARN CBAS.read.CloudMegaPageReadContext [SA:JID:1.434:TAID:TID:ANID:ODID:1:0:26:0:0] Error while pinning page number 16 with number of pages 3. stream: {file: /var/cb-cache/@analytics/v_iodevice_10/storage/partition_26/Default/Default/remote_p0yMY_volCollection_4_dzkqi/0/remote_p0yMY_volCollection_4_dzkqi/0_418_b, streamOffset: 1522705899, streamRemaining: 196526}, columnRanges:


      java.lang.IllegalStateException: Misaligned positions in /var/cb-cache/@analytics/v_iodevice_7/storage/partition_103/Default/Default/remote_DJX1u_volCollection_3_ulbow/0/remote_DJX1u_volCollection_3_ulbow/919_924_b, cloudOffset: 8380151 != requestedWriteOffset: 8488287\njava.lang.IllegalStateException: Misaligned positions in /var/cb-cache/@analytics/v_iodevice_7/storage/partition_103/Default/Default/remote_DJX1u_volCollection_3_ulbow/0/remote_DJX1u_volCollection_3_ulbow/919_924_b, cloudOffset: 8380151 != requestedWriteOffset: 8550341\

      DCP Ingestion is going up and down being slow for most of the time:

      Note: The average ingestion speed is very slow during 40-60TB which is a 3rd iteration in the test. Seems like a data loss issue.

      2024-07-12T17:01:01.952+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:1.434:TAID:TID:ANID:ODID:1:0:26:0:0] Restarting process to ensure data integrity

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty