Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.0
-
centos2 cluster
-
Untriaged
-
-
Yes
-
CX Sprint 118
Description
Build : 6.0.0-1592
Test : -test tests/analytics/test_analytics_rebalance.yml -scope tests/analytics/scope_analytics_rebalance.yml
Scale : 4
Iteration : 2nd
A step in the component system test for Analytics to remove an analytics node failed due the below mentioned error. This happened in the 2nd iteration of the test. The same step had passed in the first iteration.
Rebalance exited with reason {service_rebalance_failed,cbas,
{rebalance_failed,
}}
On 172.23.96.216,
Analytics unable to successfully rebalance 3635cd79864ca4286f1fabffe2546e85 due to 'Connection has been aborted'; see analytics log for details .
Excerpt from the analytics_error.log for this rebalance :
2018-09-01T08:12:15.456-07:00 ERRO CBAS.rebalance.Rebalance [Executor-1201:ClusterController] rebalance failed
|
org.apache.hyracks.api.exceptions.HyracksDataException: Connection has been aborted
|
at org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:69) ~[hyracks-comm.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.partitions.PipelinedPartition.nextFrame(PipelinedPartition.java:82) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.util.FrameUtils.appendToWriter(FrameUtils.java:143) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.std.connectors.PartitionDataWriter.nextFrame(PartitionDataWriter.java:131) ~[hyracks-dataflow-std.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.util.FrameUtils.appendToWriter(FrameUtils.java:185) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:266) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:290) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.std.misc.ConstantTupleSourceOperatorNodePushable.initialize(ConstantTupleSourceOperatorNodePushable.java:51) ~[hyracks-dataflow-std.jar:6.0.0-1592]
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212) ~[hyracks-api.jar:6.0.0-1592]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
|
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
|
2018-09-01T08:12:16.130-07:00 ERRO CBAS.servlet.RebalanceServlet [HttpExecutor(port:9111)-1] Rebalance 3635cd79864ca4286f1fabffe2546e85 failed
|
org.apache.hyracks.api.exceptions.HyracksDataException: Connection has been aborted
|
at org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:69) ~[hyracks-comm.jar:6.0.0-1592]
|
at org.apache.hyracks.control.nc.partitions.PipelinedPartition.nextFrame(PipelinedPartition.java:82) ~[hyracks-control-nc.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.util.FrameUtils.appendToWriter(FrameUtils.java:143) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.std.connectors.PartitionDataWriter.nextFrame(PartitionDataWriter.java:131) ~[hyracks-dataflow-std.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.util.FrameUtils.appendToWriter(FrameUtils.java:185) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:266) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:290) ~[hyracks-storage-am-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93) ~[hyracks-dataflow-common.jar:6.0.0-1592]
|
at org.apache.hyracks.dataflow.std.misc.ConstantTupleSourceOperatorNodePushable.initialize(ConstantTupleSourceOperatorNodePushable.java:51) ~[hyracks-dataflow-std.jar:6.0.0-1592]
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212) ~[hyracks-api.jar:6.0.0-1592]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
|
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
|
Marking it as a regression since this issue wasn't seen in 6.0.0-1529 (alice beta build)
Attachments
Issue Links
- causes
-
-
- Closed
-
- links to