Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
7.2.0
-
7.1.4-3631 -> 7.2.0-5322
-
Untriaged
-
Centos 64-bit
-
0
-
No
-
Analytics Sprint 19
Description
Steps to Repro
1. Run a longevity test on 7.1.4 for 4 days.
./sequoia -client 172.23.104.27:2375 -provider file:centos_pine.yml -test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_neo_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.1.4-3601 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. Upgrading to 7.2.0-5322 using online upgrade with failover/recovery strategy.
During upgrade I believe we started seeing following errors.
172.23.121.77 : analytics
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
from analytics_error.log
2023-05-05T01:42:56.093-07:00 ERRO CBAS.active.ActiveEntityEventsListener [Executor-12:ClusterController] Failed to start the entity Default/Local/WAREHOUSE
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
at org.apache.asterix.messaging.CCMessageBroker.sendSyncRequestToNCs(CCMessageBroker.java:122) ~[asterix-app-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.util.DcpUtil.requestShadowStates(DcpUtil.java:254) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.util.DcpUtil.aggregateDcpState(DcpUtil.java:204) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.coordinate(CouchbaseConnectorFactory.java:572) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.lambda$submitShadowStateFutures$4(CouchbaseConnectorFactory.java:526) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.util.BlockingUnwindFuture.lambda$submit$1(BlockingUnwindFuture.java:50) ~[cbas-common-7.1.4-3631.jar:7.1.4-3631]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
|
at java.lang.Thread.run(Thread.java:829) ~[?:?]
|
2023-05-05T01:42:56.095-07:00 ERRO CBAS.metadata.BucketEventsListener [Executor-12:ClusterController] Failed to reconnect bucket Default/Local/WAREHOUSE; attempting to recover
|
org.apache.asterix.common.exceptions.RuntimeDataException: ASX0052: Illegal state. unable to send sync message to 6307c0edad379195fcc821796ab3510a
|
at org.apache.asterix.messaging.CCMessageBroker.sendSyncRequestToNCs(CCMessageBroker.java:122) ~[asterix-app-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.util.DcpUtil.requestShadowStates(DcpUtil.java:254) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.util.DcpUtil.aggregateDcpState(DcpUtil.java:204) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.coordinate(CouchbaseConnectorFactory.java:572) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.adapter.CouchbaseConnectorFactory.lambda$submitShadowStateFutures$4(CouchbaseConnectorFactory.java:526) ~[cbas-connector-7.1.4-3631.jar:7.1.4-3631]
|
at com.couchbase.analytics.util.BlockingUnwindFuture.lambda$submit$1(BlockingUnwindFuture.java:50) ~[cbas-common-7.1.4-3631.jar:7.1.4-3631]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
|
at java.lang.Thread.run(Thread.java:829) ~[?:?]
|
I saw no issue with the upgrade but considering we added ASX keyword to our eagle eye monitoring system, it's probably of significance.
cbcollect_info attached.