Ingestion hung after the last scaling operation. Global recovery failed. Shutting down...

2024-07-19T11:06:03.578+00:00 ERRO CBAS.active.ActiveEntityEventsListener [ActiveNotificationHandler] ingestion job JID:4.3385 failed

org.apache.hyracks.api.exceptions.HyracksDataException: Connection has been aborted

        at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:70) ~[hyracks-api.jar:1.0.0-2209]

        at org.apache.hyracks.api.util.ExceptionUtils.setNodeIds(ExceptionUtils.java:70) ~[hyracks-api.jar:1.0.0-2209]

        at org.apache.hyracks.control.nc.Task.run(Task.java:398) ~[hyracks-control-nc.jar:1.0.0-2209]

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]

        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

2024-07-19T11:06:05.802+00:00 FATA CBAS.bootstrap.GlobalRecoveryManager [Executor-270:ClusterController] Global recovery failed. Shutting down...

org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.Exception: topology revision changed; was 16 now 17

        at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api.jar:1.0.0-2209]

        at com.couchbase.analytics.control.rebalance.TopologyManager.ensureTopology(TopologyManager.java:110) ~[columnar-server.jar:1.0.0-2209]

        at com.couchbase.analytics.bootstrap.AnalyticsGlobalRecoveryManager.recoverReplicas(AnalyticsGlobalRecoveryManager.java:298) ~[columnar-server.jar:1.0.0-2209]

        at com.couchbase.analytics.bootstrap.AnalyticsGlobalRecoveryManager.doRecovery(AnalyticsGlobalRecoveryManager.java:72) ~[columnar-server.jar:1.0.0-2209]

        at org.apache.asterix.hyracks.bootstrap.GlobalRecoveryManager.recover(GlobalRecoveryManager.java:120) ~[asterix-app.jar:1.0.0-2209]

        at org.apache.asterix.hyracks.bootstrap.GlobalRecoveryManager.lambda$startGlobalRecovery$0(GlobalRecoveryManager.java:96) ~[asterix-app.jar:1.0.0-2209]

        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]

        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]

        at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

Caused by: java.lang.Exception: topology revision changed; was 16 now 17

        at com.couchbase.analytics.control.rebalance.TopologyCoordinator.allSynced(TopologyCoordinator.java:134) ~[columnar-server.jar:1.0.0-2209]

        at com.couchbase.analytics.control.rebalance.TopologyCoordinator.waitForSync(TopologyCoordinator.java:102) ~[columnar-server.jar:1.0.0-2209]

        at com.couchbase.analytics.control.rebalance.TopologyCoordinator.coordinate(TopologyCoordinator.java:89) ~[columnar-server.jar:1.0.0-2209]

        at com.couchbase.analytics.control.rebalance.TopologyManager.ensureTopology(TopologyManager.java:107) ~[columnar-server.jar:1.0.0-2209]

        ... 9 more

2024-07-19T11:06:06.387+00:00 INFO CBAS.bootstrap.AnalyticsNCApplication [ShutdownHook-051705f9f49e58f33c7c99ae2511d14c] Stopping Couchbase Analytics driver

2024-07-19T11:06:06.666+00:00 ERRO CBAS.rebalance.TopologyMonitor [Topology Monitor] failed to perform storage cleanup; keep partitions [0, -1, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]