Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62848

[DCP, 40TB]: Data rollbacked to 0 for all the remote collections and no ingestion since then. Whole data is lost.

    XMLWordPrintable

Details

    Description

      JVM halts observed:

      2024-07-20T05:35:57.766+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:110:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.util.ExitUtil [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:110:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:110:0:0,5,main], interrupted true)
      		 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:111:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:114:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:57.767+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:113:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:58.398+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:06:20.721+00:00 FATA CBAS.util.ExitUtil [Executor-18595:d0ae3bb2738a1e61949b76e617764450] JVM halting with status 88 (halting thread Thread[Executor-18595:d0ae3bb2738a1e61949b76e617764450,5,main], interrupted false)
      		 
      2024-07-20T22:06:21.707+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:1:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:110:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.util.ExitUtil [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:1:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:1:0:0,5,main], interrupted true)
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:114:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:111:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.290+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:0:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.693+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:19:45.433+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:110:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:19:45.434+00:00 FATA CBAS.util.ExitUtil [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:110:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:110:0:0,5,main], interrupted true)
      		 
      2024-07-20T22:19:45.435+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:3:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:19:45.433+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:2:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:19:46.037+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:30:09.510+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:15:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:09.510+00:00 FATA CBAS.util.ExitUtil [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:15:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:15:0:0,10,main], interrupted true)
      		 
      2024-07-20T22:30:09.511+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:4:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:09.509+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:14:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:09.518+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:10:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:10.613+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-21T06:38:56.349+00:00 FATA CBAS.util.ExitUtil [Executor-3382:d0ae3bb2738a1e61949b76e617764450] JVM halting with status 88 (halting thread Thread[Executor-3382:d0ae3bb2738a1e61949b76e617764450,5,main], interrupted false)
      		 
      2024-07-21T06:38:57.661+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:110:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.util.ExitUtil [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:110:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:110:0:0,5,main], interrupted true)
      		 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:111:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:57.766+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:114:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:57.767+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:0.3670:TAID:TID:ANID:ODID:1:0:113:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T05:35:58.398+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:06:20.721+00:00 FATA CBAS.util.ExitUtil [Executor-18595:d0ae3bb2738a1e61949b76e617764450] JVM halting with status 88 (halting thread Thread[Executor-18595:d0ae3bb2738a1e61949b76e617764450,5,main], interrupted false)
      		 
      2024-07-20T22:06:21.707+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:1:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:110:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.util.ExitUtil [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:1:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:1:0:0,5,main], interrupted true)
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:114:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.288+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:111:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.290+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.8568:TAID:TID:ANID:ODID:1:0:0:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:13:35.693+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:19:45.433+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:110:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:19:45.433+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:2:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:19:45.434+00:00 FATA CBAS.util.ExitUtil [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:110:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:110:0:0,5,main], interrupted true)
      		 
      2024-07-20T22:19:45.435+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.9915:TAID:TID:ANID:ODID:1:0:3:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:19:46.037+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-20T22:30:09.510+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:15:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:09.510+00:00 FATA CBAS.util.ExitUtil [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:15:0:0] JVM halting with status 103 (halting thread Thread[SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:15:0:0,10,main], interrupted true)
      		 
      2024-07-20T22:30:09.511+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:4:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:09.509+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:14:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:09.518+00:00 FATA CBAS.runtime.DcpUpdateCallback [SA:JID:7.13916:TAID:TID:ANID:ODID:1:0:10:0:0] Restarting process to ensure data integrity
      		 
      2024-07-20T22:30:10.613+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2024-07-21T06:38:56.349+00:00 FATA CBAS.util.ExitUtil [Executor-3382:d0ae3bb2738a1e61949b76e617764450] JVM halting with status 88 (halting thread Thread[Executor-3382:d0ae3bb2738a1e61949b76e617764450,5,main], interrupted false)

      The test is rerunning on 2230 but need Devs help in debugging what's gone wrong here.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty