Data ingestion is hung due to local network error. actual count: 81553211, expected count: 1000000000 on remote_VoHss_volCollection_0_adadr collection while KV has 1B items in associated KV collection.

024-06-12T19:08:50.658+00:00 WARN CBAS.nc.Task [SA:JID:0.430:TAID:TID:ANID:ODID:1:0:14:0:0] Task failed with exception

org.apache.hyracks.api.exceptions.HyracksDataException: HYR0115: Local network error

        at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:57) ~[hyracks-api.jar:1.0.0-2134]

        at org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:115) ~[hyracks-dataflow-std.jar:1.0.0-2134]

        at org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43) ~[hyracks-dataflow-std.jar:1.0.0-2134]

        at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:424) ~[hyracks-control-nc.jar:1.0.0-2134]

        at org.apache.hyracks.control.nc.Task.run(Task.java:362) [hyracks-control-nc.jar:1.0.0-2134]

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

        Suppressed: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException                at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api.jar:1.0.0-2134]                at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:262) ~[hyracks-api.jar:1.0.0-2134]                at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:99) ~[hyracks-api.jar:1.0.0-2134]                at org.apache.hyracks.control.nc.Task.run(Task.java:372) [hyracks-control-nc.jar:1.0.0-2134]

                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]

                at java.base/java.lang.Thread.run(Thread.java:840) [?:?]        Caused by: java.lang.InterruptedException

                at java.base/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:418) ~[?:?]                at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:190) ~[?:?]                at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:245) ~[hyracks-api.jar:1.0.0-2134]                ... 5 more2024-06-12T19:08:50.659+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:1470

2024-06-12T19:08:50.660+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:14692024-06-12T19:08:50.660+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:1471

2024-06-12T19:08:50.661+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:14732024-06-12T19:08:50.661+00:00 INFO CBAS.nc.RecoveryManager [Worker:4512b6284f0da58f6b891df87fa782cf] no need to rollback as there were no operations by TxnId:14722024-06-12T19:08:50.661+00:00 WARN CBAS.active.ActiveEntityEventsListener [ActiveNotificationHandler] ingestion job JID:0.430 finished with status=(FAILURE,[Connection has been aborted]), reported runtime registrations=32, deregistrations=162024-06-12T19:08:50.662+00:00 ERRO CBAS.active.ActiveEntityEventsListener [ActiveNotificationHandler] ingestion job JID:0.430 failed

org.apache.hyracks.api.exceptions.HyracksDataException: Connection has been aborted        at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:70) ~[hyracks-api.jar:1.0.0-2134]

        at org.apache.hyracks.api.util.ExceptionUtils.setNodeIds(ExceptionUtils.java:70) ~[hyracks-api.jar:1.0.0-2134]        at org.apache.hyracks.control.nc.Task.run(Task.java:398) ~[hyracks-control-nc.jar:1.0.0-2134]

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2024-06-12 19:07:06,881 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 75233393, expected count: 1000000000

2024-06-12 19:07:19,279 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 75537889, expected count: 1000000000

2024-06-12 19:07:31,529 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 77271745, expected count: 1000000000

2024-06-12 19:07:43,608 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 77823667, expected count: 1000000000

2024-06-12 19:07:55,867 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 78380298, expected count: 1000000000

2024-06-12 19:08:12,423 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 80231894, expected count: 1000000000

2024-06-12 19:08:27,628 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 80561637, expected count: 1000000000

2024-06-12 19:09:10,779 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553053, expected count: 1000000000

2024-06-12 19:09:21,496 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:09:31,790 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:09:42,117 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:09:52,459 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:10:02,792 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:10:13,128 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:10:23,950 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:10:34,368 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:10:44,732 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:10:55,053 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:11:05,405 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000

2024-06-12 19:11:15,792 | test  | DEBUG   | MainThread | [CbasUtil:wait_for_ingestion:222] dataset: remote_VoHss_volCollection_0_adadr, status: SUCCESS, actual count: 81553211, expected count: 1000000000