Details
-
Bug
-
Resolution: Resolved
-
Critical
-
Columnar 1.0.0
-
1.0.0-2209
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 46
Description
Workload -
| Type | Number of collections | Number of items in millions | Total count in millions |
|---|---|---|---|
| Remote | 80 | 75 | 6000 |
| Standalone | 50 | 8 | 4000* |
| Kafka | 5 | 33.5 | 165 |
*Some standalone collections have 8 mil and some have multiples of 8 million items. The total doc count is 4000 million ( 4 billion) items.
Number of links = 6 ( 2 remote + 2 external + 2 kafka). 1 remote link and 1 kafka link is active.
OOM seen on node 003
2024-07-15T17:45:09.751+00:00 INFO CBAS.cbas updating svc-da-node-028.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:11.837+00:00 INFO CBAS.cbas updating svc-da-node-031.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:13.646+00:00 WARN CBAS.cbas got error updating svc-da-node-019.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver: Post "https://svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:9110/analytics/internal/cbas/refreshAuth": context deadline exceeded (Client.Timeout exceeded while awaiting headers); will retry |
2024-07-15T17:45:14.646+00:00 INFO CBAS.cbas updating svc-da-node-019.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:19.701+00:00 WARN CBAS.cbas got error updating svc-da-node-021.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver: Post "https://svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:9110/analytics/internal/cbas/refreshAuth": context deadline exceeded (Client.Timeout exceeded while awaiting headers); will retry |
2024-07-15T17:45:20.701+00:00 INFO CBAS.cbas updating svc-da-node-021.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:24.739+00:00 WARN CBAS.cbas got error updating svc-da-node-026.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver: Post "https://svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:9110/analytics/internal/cbas/refreshAuth": context deadline exceeded (Client.Timeout exceeded while awaiting headers); will retry |
2024-07-15T17:45:25.739+00:00 INFO CBAS.cbas updating svc-da-node-026.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:30.812+00:00 WARN CBAS.cbas got error updating svc-da-node-032.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver: Post "https://svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:9110/analytics/internal/cbas/refreshAuth": context deadline exceeded (Client.Timeout exceeded while awaiting headers); will retry |
2024-07-15T17:45:31.812+00:00 INFO CBAS.cbas updating svc-da-node-032.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:35.595+00:00 WARN CBAS.cbas got error updating svc-da-node-017.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver: Post "https://svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:9110/analytics/internal/cbas/refreshAuth": context deadline exceeded (Client.Timeout exceeded while awaiting headers); will retry |
2024-07-15T17:45:36.596+00:00 INFO CBAS.cbas updating svc-da-node-017.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:45:41.889+00:00 WARN CBAS.cbas got error updating svc-da-node-031.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver: Post "https://svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:9110/analytics/internal/cbas/refreshAuth": context deadline exceeded (Client.Timeout exceeded while awaiting headers); will retry |
2024-07-15T17:45:42.890+00:00 INFO CBAS.cbas updating svc-da-node-031.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com:8095 httpService creds on driver |
2024-07-15T17:47:20.887+00:00 INFO CBAS.work.AbortTasksWork [Worker:4b7817e434b0e84cb7c0fb1dd6961bff] Aborting Tasks: JID:0.4479:[TAID:TID:ANID:ODID:2:1:32:0, TAID:TID:ANID:ODID:2:1:33:0, TAID:TID:ANID:ODID:2:1:34:0, TAID:TID:ANID:ODID:2:1:35:0, TAID:TID:ANID:ODID:5:1:32:0, TAID:TID:ANID:ODID:2:1:36:0, TAID:TID:ANID:ODID:5:1:33:0, TAID:TID:ANID:ODID:2:1:37:0, TAID:TID:ANID:ODID:5:1:34:0, TAID:TID:ANID:ODID:2:1:38:0, TAID:TID:ANID:ODID:5:1:35:0, TAID:TID:ANID:ODID:2:1:39:0, TAID:TID:ANID:ODID:5:1:36:0, TAID:TID:ANID:ODID:2:1:40:0, TAID:TID:ANID:ODID:5:1:37:0, TAID:TID:ANID:ODID:2:1:41:0, TAID:TID:ANID:ODID:5:1:38:0, TAID:TID:ANID:ODID:2:1:42:0, TAID:TID:ANID:ODID:5:1:39:0, TAID:TID:ANID:ODID:1:1:44:0, TAID:TID:ANID:ODID:2:1:43:0, TAID:TID:ANID:ODID:5:1:40:0, TAID:TID:ANID:ODID:2:1:44:0, TAID:TID:ANID:ODID:5:1:41:0, TAID:TID:ANID:ODID:1:1:46:0, TAID:TID:ANID:ODID:2:1:45:0, TAID:TID:ANID:ODID:5:1:42:0, TAID:TID:ANID:ODID:2:1:46:0, TAID:TID:ANID:ODID:5:1:43:0, TAID:TID:ANID:ODID:2:1:47:0, TAID:TID:ANID:ODID:5:1:44:0, TAID:TID:ANID:ODID:5:1:45:0, TAID:TID:ANID:ODID:1:1:50:0, TAID:TID:ANID:ODID:5:1:46:0, TAID:TID:ANID:ODID:5:1:47:0, TAID:TID:ANID:ODID:1:1:52:0] |
Terminating due to java.lang.OutOfMemoryError: Java heap space
|
cbcollect ->
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-001.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-002.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-003.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-004.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-005.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-006.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-007.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-008.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-009.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-010.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-011.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-012.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-013.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-014.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-015.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-016.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-017.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-018.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-019.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-020.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-021.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-022.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-023.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-024.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-025.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-026.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-027.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-028.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-029.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-030.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-031.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnar15July/collectinfo-2024-07-15T182549-ns_1%40svc-da-node-032.twi3gef5x8hk6evi.sandbox.nonprod-project-avengers.com.zip
Attachments
Issue Links
- causes
-
-
- Closed
-