Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62642

[System Test] Terminating due to java.lang.OutOfMemoryError: Java heap space error seen

    XMLWordPrintable

Details

    • Bug
    • Resolution: Resolved
    • Critical
    • Columnar 1.0.0
    • Columnar 1.0.0
    • analytics
    • 1.0.0-2200 Columnar 4-node cluster ( 16 VCPUs + 64 GB)
    • Untriaged
    • 0
    • Unknown
    • Analytics Sprint 46

    Description

      There have been out of memory errors seen at these timestamps -

      On node 001

      ./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3755-2024-07-09T04:41:48.661+00:00 INFO CBAS.adapter.CouchbaseConnector [cbas:linkHronvsfg:default1:fd9569d26a76b55d456fdd1f30044979:16 idle connection watchdog] notifying CC on 2 idle stream(s) thresholds met
      ./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3756-2024-07-09T04:42:32.145+00:00 INFO CBAS.server.QueryServiceServlet [HttpExecutor(port:18095)-2] handleRequest: uuid=4fbea5fa-39a9-4827-aa13-9724fc48896b, clientContextID=null, {"host":"cb.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com:18095","path":"/analytics/service","statement":"<ud>SELECT c.category, p.product_name, p.price, AVG(r.product_rating.rating_value) AS avg_rating, COUNT(r.product_rating) AS num_reviews FROM `Database8OpxKbOqW`.`scope0qqwikVtk`.`standalonemWFwCxyr` p LEFT JOIN (SELECT product_name, product_rating FROM `Database8OpxKbOqW`.`scope0qqwikVtk`.`standalonemWFwCxyr`) r ON p.product_name = r.product_name JOIN (SELECT product_category, MAX(price) AS max_price FROM `Database8OpxKbOqW`.`scope0qqwikVtk`.`standalonemWFwCxyr` GROUP BY product_category) c ON p.product_category = c.product_category AND p.price = c.max_price GROUP BY c.category, p.product_name, p.price ORDER BY c.category, p.price DESC limit 1000;</ud>","pretty":false,"mode":"immediate","clientContextID":null,"clientType":"ASTERIX","dataverse":null,"format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"profile":"counts","signature":true,"multiStatement":true,"parseOnly":false,"readOnly":false,"maxWarnings":0,"sqlCompat":false,"source":null,"scanConsistency":null,"scanWait":null}
      ./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log:3757:Terminating due to java.lang.OutOfMemoryError: Java heap space
      ./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3758-2024-07-09T04:43:35.508+00:00 WARN CBAS.cbas analytics driver has exited w/ exit status 3
      ./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3759-2024-07-09T04:43:35.529+00:00 INFO CBAS.cbas analytics driver exited; restarting
      

      On node 002

      ./cbcollect_info_ns_1@svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-4116-2024-07-09T04:42:27.798+00:00 INFO CBAS.messaging.NCMessageBroker [Worker:eebcfc3cdc98804aed77d6dad6f0c9c5] Received message: ExecuteStatementResponseMessage(id=205, uuid=aef7b919-fd1c-4264-96f8-ea6d276033bc, clientContextId=null): 0 characters
      ./cbcollect_info_ns_1@svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-4117-2024-07-09T04:43:45.899+00:00 INFO CBAS.adapter.CouchbaseConnector [cbas:linkHronvsfg:default1:eebcfc3cdc98804aed77d6dad6f0c9c5:32 idle connection watchdog] notifying CC on 2 idle stream(s) thresholds met
      ./cbcollect_info_ns_1@svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log:4118:Terminating due to java.lang.OutOfMemoryError: Java heap space
      

      On node 004

      ./cbcollect_info_ns_1@svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-13416-2024-07-09T04:44:32.894+00:00 INFO CBAS.cbas updating svc-da-node-003.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver
      ./cbcollect_info_ns_1@svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-13417-2024-07-09T04:44:32.947+00:00 INFO CBAS.cbas updating svc-da-node-003.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver
      ./cbcollect_info_ns_1@svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log:13418:Terminating due to java.lang.OutOfMemoryError: Java heap space
      

      Workload -

      Type Number of collections Number of items in millions Total count in millions
      Remote 80 75 6000
      Standalone 50 8 4000*

      Some standalone collections have 8 mil and some have multiples of 8 million items. The total doc count is 4000 million ( 4 billion) items.
      Number of links = 5 ( 3 remote + 2 external)

      This was before the fix for https://issues.couchbase.com/browse/MB-62597 was merged, but the unexpected number of bytes error only start at this timestamp(i.e after the OOM errors) -

      2024-07-09T04:47:12
      

      There are no Copy from Parquet files as far as I can see so it can't be https://couchbasecloud.atlassian.net/browse/AV-71781 .

      cbcollect ->

      https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-003.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              pavan.pb Pavan PB
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty