Details
-
Bug
-
Resolution: Resolved
-
Critical
-
Columnar 1.0.0
-
1.0.0-2200 Columnar 4-node cluster ( 16 VCPUs + 64 GB)
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 46
Description
There have been out of memory errors seen at these timestamps -
On node 001
./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3755-2024-07-09T04:41:48.661+00:00 INFO CBAS.adapter.CouchbaseConnector [cbas:linkHronvsfg:default1:fd9569d26a76b55d456fdd1f30044979:16 idle connection watchdog] notifying CC on 2 idle stream(s) thresholds met |
./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3756-2024-07-09T04:42:32.145+00:00 INFO CBAS.server.QueryServiceServlet [HttpExecutor(port:18095)-2] handleRequest: uuid=4fbea5fa-39a9-4827-aa13-9724fc48896b, clientContextID=null, {"host":"cb.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com:18095","path":"/analytics/service","statement":"<ud>SELECT c.category, p.product_name, p.price, AVG(r.product_rating.rating_value) AS avg_rating, COUNT(r.product_rating) AS num_reviews FROM `Database8OpxKbOqW`.`scope0qqwikVtk`.`standalonemWFwCxyr` p LEFT JOIN (SELECT product_name, product_rating FROM `Database8OpxKbOqW`.`scope0qqwikVtk`.`standalonemWFwCxyr`) r ON p.product_name = r.product_name JOIN (SELECT product_category, MAX(price) AS max_price FROM `Database8OpxKbOqW`.`scope0qqwikVtk`.`standalonemWFwCxyr` GROUP BY product_category) c ON p.product_category = c.product_category AND p.price = c.max_price GROUP BY c.category, p.product_name, p.price ORDER BY c.category, p.price DESC limit 1000;</ud>","pretty":false,"mode":"immediate","clientContextID":null,"clientType":"ASTERIX","dataverse":null,"format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"profile":"counts","signature":true,"multiStatement":true,"parseOnly":false,"readOnly":false,"maxWarnings":0,"sqlCompat":false,"source":null,"scanConsistency":null,"scanWait":null} |
./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log:3757:Terminating due to java.lang.OutOfMemoryError: Java heap space |
./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3758-2024-07-09T04:43:35.508+00:00 WARN CBAS.cbas analytics driver has exited w/ exit status 3 |
./cbcollect_info_ns_1@svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-3759-2024-07-09T04:43:35.529+00:00 INFO CBAS.cbas analytics driver exited; restarting |
On node 002
./cbcollect_info_ns_1@svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-4116-2024-07-09T04:42:27.798+00:00 INFO CBAS.messaging.NCMessageBroker [Worker:eebcfc3cdc98804aed77d6dad6f0c9c5] Received message: ExecuteStatementResponseMessage(id=205, uuid=aef7b919-fd1c-4264-96f8-ea6d276033bc, clientContextId=null): 0 characters |
./cbcollect_info_ns_1@svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-4117-2024-07-09T04:43:45.899+00:00 INFO CBAS.adapter.CouchbaseConnector [cbas:linkHronvsfg:default1:eebcfc3cdc98804aed77d6dad6f0c9c5:32 idle connection watchdog] notifying CC on 2 idle stream(s) thresholds met |
./cbcollect_info_ns_1@svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log:4118:Terminating due to java.lang.OutOfMemoryError: Java heap space |
On node 004
./cbcollect_info_ns_1@svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-13416-2024-07-09T04:44:32.894+00:00 INFO CBAS.cbas updating svc-da-node-003.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
./cbcollect_info_ns_1@svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log-13417-2024-07-09T04:44:32.947+00:00 INFO CBAS.cbas updating svc-da-node-003.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
./cbcollect_info_ns_1@svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com_20240709-055324/ns_server.analytics_info.log:13418:Terminating due to java.lang.OutOfMemoryError: Java heap space |
Workload -
Type | Number of collections | Number of items in millions | Total count in millions |
---|---|---|---|
Remote | 80 | 75 | 6000 |
Standalone | 50 | 8 | 4000* |
Some standalone collections have 8 mil and some have multiples of 8 million items. The total doc count is 4000 million ( 4 billion) items.
Number of links = 5 ( 3 remote + 2 external)
This was before the fix for https://issues.couchbase.com/browse/MB-62597 was merged, but the unexpected number of bytes error only start at this timestamp(i.e after the OOM errors) -
2024-07-09T04:47:12 |
There are no Copy from Parquet files as far as I can see so it can't be https://couchbasecloud.atlassian.net/browse/AV-71781 .
cbcollect ->
https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-001.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-002.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-003.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar8Jul/collectinfo-2024-07-09T055323-ns_1%40svc-da-node-004.4h4zh8vtyczmkulg.sandbox.nonprod-project-avengers.com.zip
Attachments
Issue Links
- relates to
-
MB-62637 CC running out of heap space
-
- Closed
-