Loading...

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.2.1, 7.1.5
Affects Version/s: 7.1.5
Component/s: analytics
Labels:
Environment:
Enterprise Edition 7.1.5 build 3876

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.104.176.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.104.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.104.249.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.0.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.210.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.38.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.39.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.91.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.106.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.106.37.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.107.142.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.107.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.107.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.129.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.132.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.138.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.139.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.140.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.143.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.144.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.145.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.146.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.34.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.61.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.97.179.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.104.176.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.104.216.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.104.249.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.0.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.210.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.38.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.39.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.105.91.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.106.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.106.37.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.106.54.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.107.142.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.107.236.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.107.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.129.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.132.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.136.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.138.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.139.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.140.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.143.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.144.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.145.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.146.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.34.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.108.61.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1689552901/collectinfo-2023-07-17T001502-ns_1%40172.23.97.179.zip
Story Points:
0
Is this a Regression?:
Unknown
Sprint:
Analytics Sprint 22

Description

QE Test

./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_neo_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.1.5-3876 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true

Rebalance Operation
2023-07-16T13:04:38.787-07:00, ns_orchestrator:0:info:message(ns_1@172.23.108.139) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.176','ns_1@172.23.104.249',
'ns_1@172.23.105.0','ns_1@172.23.105.134',
'ns_1@172.23.105.210','ns_1@172.23.105.38',
'ns_1@172.23.105.39','ns_1@172.23.105.91',
'ns_1@172.23.106.32','ns_1@172.23.106.37',
'ns_1@172.23.106.54','ns_1@172.23.107.142',
'ns_1@172.23.107.236','ns_1@172.23.107.25',
'ns_1@172.23.108.129','ns_1@172.23.108.132',
'ns_1@172.23.108.134','ns_1@172.23.108.136',
'ns_1@172.23.108.138','ns_1@172.23.108.139',
'ns_1@172.23.108.140','ns_1@172.23.108.141',
'ns_1@172.23.108.143','ns_1@172.23.108.144',
'ns_1@172.23.108.145','ns_1@172.23.108.146',
'ns_1@172.23.108.148','ns_1@172.23.108.34',
'ns_1@172.23.108.61','ns_1@172.23.97.179'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 5f0b5dcab2231f99ff9803b658c74ceb

Rebalance Failure
2023-07-16T13:10:09.549-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.108.139) - Rebalance exited with reason {service_rebalance_failed,cbas,
{worker_died,
{'EXIT',<0.21785.3047>,
{rebalance_failed,
{service_error,
<<"Rebalance 0fa3bc61bb63b43da60bf215d367d0ee failed: timed out waiting for all nodes to join & cluster active (missing nodes: [172.23.104.176:8091 (9817349cfbed040a46788e6267e5970a), 172.23.105.0:8091 (e4b5dd89d35841b1163c1c9b33a71033)], state: UNUSABLE)">>}}}}}.
Rebalance Operation Id = 5f0b5dcab2231f99ff9803b658c74ceb

Rebalance operations are failing continuously due to this same error.

grep "java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active" ns_server.analytics_info.log | wc -l

NOTE

We did not see this issue on previous 7.1.5 longevity runs, last run on 7.1.5-3858 build.

Attachments

Issue Links

links to

ASTERIXDB-3226

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[System Test] Analytics rebalance is failing continously with timed out waiting for all nodes to join & cluster active (missing nodes: [9817349cfbed040a46788e6267e5970a, e4b5dd89d35841b1163c1c9b33a71033], state: UNUSABLE) error

Details

Description

NOTE

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty