Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62127

[Crash/Recovery]: Ingestion is hung forever on crashing(sigkill) cbas process periodically with an interval of 20s for a while.

    XMLWordPrintable

Details

    Description

      1. Load 5B items in remote couchbase cluster
      2. Create remote collection and start ingesting data
      3. After few hours while ingestion start a kill -9 on one of the cbas node
      4. Ingestion is hung at 20661550 items. 

        2024-06-01 04:20:06,368 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:20:25,119 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:20:44,579 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:21:05,322 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:21:25,184 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:21:44,894 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:22:05,977 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:22:26,158 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000
        		 
        2024-06-01 04:22:44,786 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_Wu8tU_volCollection_0_kapoj, status: SUCCESS, actual count: 20661550, expected count: 1000000000

      1. After a while analytics is crashing in a loop on its own and couldn't recover. Service 'cbas' exited with status 137.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty