Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.6.2
-
7.0.0-5295, 7.0.0-155879
-
Untriaged
-
Centos 64-bit
-
-
1
-
No
-
CX Sprint 252, CX Sprint 253
Description
Test Scenario –
Ingest approximately 4 TB of data from Remote KV cluster to Local CBAS cluster.
Infra –
Remote KV cluster:
|
Number of nodes – 20
|
RAM allocated to KV per Node – 22.2 GB
|
Disk allocated per node – 25 GB
|
Number of buckets – 1
|
Number of scopes/bucket – 1
|
Number of collections/scope – 1
|
Number of items – 130,197,370
|
Total size of the data – 63.8 GB
|
Local CBAS cluster:
|
Number of nodes – 4
|
RAM allocated to CBAS per Node – 223 GB
|
Disk allocated per node – 10 TB
|
Test Setup-
Number of dataverses – 5
|
Number of Remote links – 12
|
Number of datasets – 12 (1 dataset per link)
|
Observations –
RAM usage was hovering around 85%
|
Analytics console on WebUI was not usable.
|
Any http request to CBAS was executing indefinitely or was giving Socket error.
|
Following logs were seen in analytics_info.log
2021-06-14T07:37:31.718-07:00 WARN CBAS.dataflow.FeedRecordDataFlowController [SAO:JID:0.16:TAID:TID:ANID:ODID:4:0:10:0:(dv1/link1/testBucket)[10]:BO] Exception during ingestion org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api.jar:7.0.0-5295] at org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:91) ~[hyracks-comm.jar:7.0.0-5295] at org.apache.hyracks.control.nc.partitions.PipelinedPartition.nextFrame(PipelinedPartition.java:82) ~[hyracks-control-nc.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressFrameTupleAppender.forward(ProgressFrameTupleAppender.java:184) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressFrameTupleAppender.write(ProgressFrameTupleAppender.java:162) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressFrameTupleAppender.flush(ProgressFrameTupleAppender.java:194) ~[cbas-connector.jar:7.0.0-5295] at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:117) ~[hyracks-dataflow-common.jar:7.0.0-5295] at com.couchbase.analytics.runtime.ProgressPartitionDataWriter.flush(ProgressPartitionDataWriter.java:230) ~[cbas-connector.jar:7.0.0-5295] at org.apache.hyracks.dataflow.std.base.AbstractReplicateOperatorDescriptor$ReplicatorMaterializerActivityNode$1.flush(AbstractReplicateOperatorDescriptor.java:149) ~[hyracks-dataflow-std.jar:7.0.0-5295] at org.apache.hyracks.api.comm.IFrameAppender.flush(IFrameAppender.java:68) ~[hyracks-api.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.TupleForwarder.flush(TupleForwarder.java:47) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.AbstractFeedDataFlowController.flush(AbstractFeedDataFlowController.java:50) ~[asterix-external-data.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.flushCurrentFrame(CouchbaseConnector.java:1027) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.handled(CouchbaseConnector.java:776) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.pollNextMessage(CouchbaseConnector.java:692) ~[cbas-connector.jar:7.0.0-5295] at com.couchbase.analytics.adapter.CouchbaseConnector.next(CouchbaseConnector.java:679) ~[cbas-connector.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.next(FeedRecordDataFlowController.java:139) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:88) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:41) ~[asterix-external-data.jar:7.0.0-5295] at org.apache.asterix.common.external.IDataSourceAdapter.start(IDataSourceAdapter.java:75) ~[asterix-common.jar:7.0.0-5295] at com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:52) ~[cbas-connector.jar:7.0.0-5295] at org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:102) ~[asterix-active.jar:7.0.0-5295] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:227) ~[hyracks-api.jar:7.0.0-5295] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) ~[?:?] at java.lang.Object.wait(Unknown Source) ~[?:?] at org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:85) ~[hyracks-comm.jar:7.0.0-5295] ... 25 more
|
2021-06-14T07:37:31.720-07:00 WARN CBAS.collectors.NonDeterministicChannelReader [SA:JID:0.20:TAID:TID:ANID:ODID:1:0:54:0:0] Sender failed.. returning silently
|
Attachments
Issue Links
- Clones
-
MB-46935 [Volume Test] Data ingestion is not happening when trying to ingest large data using multiple remote links (= 12 links)
-
- Closed
-
Gerrit Reviews
For Gerrit Dashboard: MB-47042 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
156309,4 | MB-47042: [BP 6.6.3] do not round-robin vbuckets across partitions, to minimize KV connections | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
157198,4 | MB-47042: [BP 6.6.3]: fix partition scheduling issue | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |