Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62461

[System Test] java.lang.OutOfMemoryError: Java heap space seen during ingestion via Kafka links

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • Columnar 1.0.0
    • Columnar 1.0.0
    • analytics
    • Columnar Edition 1.0.0 build 2172
    • Untriaged
    • 0
    • Unknown
    • Analytics Sprint 44, Analytics Sprint 45, Analytics Sprint 46

    Description

      It's a 4-node cluster (16 VCPUs + 64 GB).

      Seen on node 002- 

      2024-06-25T07:00:53.660+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11080:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.660+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11075:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.660+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11081:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.660+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11073:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.660+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11078:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.661+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11079:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.661+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11074:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:53.661+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-11076:366d39b76e6b79e287b27e683fc2a51d] Data Polling started...
      		 
      2024-06-25T07:00:55.198+00:00 INFO CBAS.bootstrap.ClusterMonitor [linkyPpdoUMQ cluster monitor] interrupted during pool stream; will retry
      		 
      2024-06-25T07:00:55.198+00:00 INFO CBAS.bootstrap.ClusterMonitor [linkEhWaXEIm cluster monitor] interrupted during pool stream; will retry
      		 
      2024-06-25T07:00:55.198+00:00 INFO CBAS.bootstrap.ClusterMonitor [linkLGikBFPE cluster monitor] interrupted during pool stream; will retry
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (2)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (3)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (4)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (5)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (6)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (7)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (10)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (11)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (12)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (13)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (14)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (15)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (18)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (19)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (20)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (21)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (22)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (23)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (26)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (27)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (28)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (29)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (30)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (31)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (34)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (35)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (36)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (37)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (38)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (39)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (42)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (365), Partition (43)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (2)
      		 
      2024-06-25T07:00:57.306+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (3)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (4)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (5)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (6)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (7)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (10)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (11)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (12)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (13)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (14)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (15)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (18)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (19)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (20)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (21)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (22)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (23)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (26)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (27)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (28)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (29)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (30)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (31)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (34)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (35)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (36)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (37)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (38)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (39)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (42)
      		 
      2024-06-25T07:00:57.307+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (368), Partition (43)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (2)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (3)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (4)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (5)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (6)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (7)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (10)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (11)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (12)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (13)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (14)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (15)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (18)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (19)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (20)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (21)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (22)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (23)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (26)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (27)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (28)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (29)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (30)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (31)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (34)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (35)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (36)
      		 
      2024-06-25T07:00:57.308+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (37)
      		 
      2024-06-25T07:00:57.309+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (38)
      		 
      2024-06-25T07:00:57.309+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (39)
      		 
      2024-06-25T07:00:57.309+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (42)
      		 
      2024-06-25T07:00:57.309+00:00 INFO CBAS.context.DatasetLifecycleManager [Checkpoint Thread (178)] Async flushing Dataset (373), Partition (43)
      		 
      2024-06-25T07:00:57.826+00:00 INFO CBAS.bootstrap.ClusterMonitor [linkYvZjVsVp cluster monitor] interrupted during pool stream; will retry
      		 
      Terminating due to java.lang.OutOfMemoryError: Java heap space
      		 
      2024-06-25T07:01:06.852+00:00 WARN CBAS.cbas analytics driver has exited w/ exit status 3

      cbcollect ->

      https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun24/collectinfo-2024-06-25T070705-ns_1%40svc-da-node-001.nxp9u2dolhjxwjtf.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun24/collectinfo-2024-06-25T070705-ns_1%40svc-da-node-002.nxp9u2dolhjxwjtf.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun24/collectinfo-2024-06-25T070705-ns_1%40svc-da-node-003.nxp9u2dolhjxwjtf.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun24/collectinfo-2024-06-25T070705-ns_1%40svc-da-node-004.nxp9u2dolhjxwjtf.sandbox.nonprod-project-avengers.com.zip

      Attachments

        Issue Links

          relates to

          Task - A task that needs to be done. MB-62475 Limit the number of links

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavan.pb Pavan PB
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty