Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Columnar 1.0.0
-
Columnar Edition 1.0.0 build 2134
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 44
Description
On continuing using the same cluster as in MB-62299 I create 5 new remote collections and the ingestion started well but it is stuck at some point and never proceeded from there on...
024-06-12T19:08:50.658+00:00 WARN CBAS.nc.Task [SA:JID:0.430:TAID:TID:ANID:ODID:1:0:14:0:0] Task failed with exception
|
org.apache.hyracks.api.exceptions.HyracksDataException: HYR0115: Local network error
|
at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:57) ~[hyracks-api.jar:1.0.0-2134]
|
at org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:115) ~[hyracks-dataflow-std.jar:1.0.0-2134]
|
at org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43) ~[hyracks-dataflow-std.jar:1.0.0-2134]
|
at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:424) ~[hyracks-control-nc.jar:1.0.0-2134]
|
at org.apache.hyracks.control.nc.Task.run(Task.java:362) [hyracks-control-nc.jar:1.0.0-2134]
|
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
|
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
|
Suppressed: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api.jar:1.0.0-2134] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:262) ~[hyracks-api.jar:1.0.0-2134] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:99) ~[hyracks-api.jar:1.0.0-2134] at org.apache.hyracks.control.nc.Task.run(Task.java:372) [hyracks-control-nc.jar:1.0.0-2134]
|
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
|
at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: java.lang.InterruptedException
|
at java.base/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418) ~[?:?] at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:190) ~[?:?] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:245) ~[hyracks-api.jar:1.0.0-2134] ... 5 more2024-06-12T19:08:50.659+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:1470
|
2024-06-12T19:08:50.660+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:14692024-06-12T19:08:50.660+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:1471
|
2024-06-12T19:08:50.661+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:14732024-06-12T19:08:50.661+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:14722024-06-12T19:08:50.661+00:00 WARN CBAS.active.ActiveEntityEventsListener [ActiveNotificationHandler] ingestion job JID:0.430 finished with status=(FAILURE,[Connection has been aborted]), reported runtime registrations=32, deregistrations=162024-06-12T19:08:50.662+00:00 ERRO CBAS.active.ActiveEntityEventsListener [ActiveNotificationHandler] ingestion job JID:0.430 failed
|
org.apache.hyracks.api.exceptions.HyracksDataException: Connection has been aborted at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:70) ~[hyracks-api.jar:1.0.0-2134]
|
at org.apache.hyracks.api.util.ExceptionUtils.setNodeIds(ExceptionUtils.java:70) ~[hyracks-api.jar:1.0.0-2134] at org.apache.hyracks.control.nc.Task.run(Task.java:398) ~[hyracks-control-nc.jar:1.0.0-2134]
|
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
|
IngestionProgress |
2024-06-12 19:07:06,881 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 75233393, expected count: 1000000000
|
2024-06-12 19:07:19,279 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 75537889, expected count: 1000000000
|
2024-06-12 19:07:31,529 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 77271745, expected count: 1000000000
|
2024-06-12 19:07:43,608 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 77823667, expected count: 1000000000
|
2024-06-12 19:07:55,867 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 78380298, expected count: 1000000000
|
2024-06-12 19:08:12,423 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 80231894, expected count: 1000000000
|
2024-06-12 19:08:27,628 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 80561637, expected count: 1000000000
|
2024-06-12 19:09:10,779 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553053, expected count: 1000000000
|
2024-06-12 19:09:21,496 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:09:31,790 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:09:42,117 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:09:52,459 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:10:02,792 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:10:13,128 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:10:23,950 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:10:34,368 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:10:44,732 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:10:55,053 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:11:05,405 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
2024-06-12 19:11:15,792 | test | DEBUG | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000
|
Attachments
Issue Links
- links to