Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48415

Unexpected halt during system tests on ingestion suspension

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No
    • CX Sprint 262

    Description

      Encountered during system tests inĀ MB-48345. Node 172.23.104.21 halted due to a timeout waiting for ingestion to be suspended.

      2021-09-06 01:09:08,827 - systestmon - INFO - --+--+--+--+-- Parsing logs for analytics component looking for fata --+--+--+--+--
      2021-09-06 01:09:09,482 - systestmon - WARNING - *** 2 occurences of fata keyword found on 172.23.104.21 ***
      2021-09-06 01:09:09,483 - systestmon - DEBUG - 2021-09-06T00:52:09.922-07:00 FATA CBAS.util.ExitUtil [Executor-75:ClusterController : WaitForCompletionForJobId: JID:0.1844] JVM halting with status 22 (halting thread Thread[Executor-75:ClusterController : WaitForCompletionForJobId: JID:0.1844,10,main], interrupted false)
      2021-09-06 01:09:09,483 - systestmon - DEBUG - 2021-09-06T00:52:10.276-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: 
      2021-09-06 01:09:12,518 - systestmon - WARNING - There have been more occurences of keyword fata in the logs since the last iteration. Hence performing a cbcollect.
      

      The issue can happen when some node in the cluster fails while we are trying to suspend ingestion due to a rebalance or a DDL. While the ingestion job actually fails, we keep waiting for the ingestion state to be "suspended" but the state will actually be "temporary failed" due to the node failure.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            umang.agrawal Umang added a comment -

            Verified with 7.0.2-6678

            umang.agrawal Umang added a comment - Verified with 7.0.2-6678

            People

              murtadha.hubail Murtadha Hubail
              murtadha.hubail Murtadha Hubail
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty