[BP 6.6.3] Data ingestion is not happening when trying to ingest large data using multiple remote links (= 12 links)

Remote KV cluster:

Number of nodes – 20

RAM allocated to KV per Node – 22.2 GB

Disk allocated per node – 25 GB

Number of buckets – 1

Number of scopes/bucket – 1

Number of collections/scope – 1

Number of items – 130,197,370

Total size of the data – 63.8 GB

Local CBAS cluster:

Number of nodes – 4

RAM allocated to CBAS per Node – 223 GB

Disk allocated per node – 10 TB

Number of dataverses – 5

Number of Remote links – 12

Number of datasets – 12 (1 dataset per link)

RAM usage was hovering around 85%

Analytics console on WebUI was not usable.

Any http request to CBAS was executing indefinitely or was giving Socket error.

2021-06-14T07:37:31.718-07:00 WARN CBAS.dataflow.FeedRecordDataFlowController [SAO:JID:0.16:TAID:TID:ANID:ODID:4:0:10:0:(dv1/link1/testBucket)[10]:BO] Exception during ingestion org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api.jar:7.0.0-5295] at org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:91) ~[hyracks-comm.jar:7.0.0-5295] at org.apache.hyracks.control.nc.partitions.PipelinedPartition.nextFrame(PipelinedPartition.java:82) ~[hyracks-control-nc.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressFrameTupleAppender.forward(ProgressFrameTupleAppender.java:184) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressFrameTupleAppender.write(ProgressFrameTupleAppender.java:162) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressFrameTupleAppender.flush(ProgressFrameTupleAppender.java:194) ~[cbas-connector.jar:7.0.0-5295] at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:117) ~[hyracks-dataflow-common.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressPartitionDataWriter.flush(ProgressPartitionDataWriter.java:230) ~[cbas-connector.jar:7.0.0-5295] at org.apache.hyracks.dataflow.std.base.AbstractReplicateOperatorDescriptor$ReplicatorMaterializerActivityNode$1.flush(AbstractReplicateOperatorDescriptor.java:149) ~[hyracks-dataflow-std.jar:7.0.0-5295] at org.apache.hyracks.api.comm.IFrameAppender.flush(IFrameAppender.java:68) ~[hyracks-api.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.TupleForwarder.flush(TupleForwarder.java:47) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.AbstractFeedDataFlowController.flush(AbstractFeedDataFlowController.java:50) ~[asterix-external-data.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.flushCurrentFrame(CouchbaseConnector.java:1027) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.handled(CouchbaseConnector.java:776) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.pollNextMessage(CouchbaseConnector.java:692) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.next(CouchbaseConnector.java:679) ~[cbas-connector.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.next(FeedRecordDataFlowController.java:139) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:88) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:41) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.common.external.IDataSourceAdapter.start(IDataSourceAdapter.java:75) ~[asterix-common.jar:7.0.0-5295] at com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:52) ~[cbas-connector.jar:7.0.0-5295] at org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:102) ~[asterix-active.jar:7.0.0-5295] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:227) ~[hyracks-api.jar:7.0.0-5295] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) ~[?:?] at java.lang.Object.wait(Unknown Source) ~[?:?] at org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:85) ~[hyracks-comm.jar:7.0.0-5295] ... 25 more 

2021-06-14T07:37:31.720-07:00 WARN CBAS.collectors.NonDeterministicChannelReader [SA:JID:0.20:TAID:TID:ANID:ODID:1:0:54:0:0] Sender failed.. returning silently