[System Test] :- Rebalances fail with "Analytics Service unable to successfully rebalance 7abd22cdd2556f30ca4e515af3920dcf due to 'java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active (missing nodes: "

./sequoia -client 172.23.104.27:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 2 -repeat 0 -log_level 0 -version 7.6.0-1845 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true

Starting rebalance, KeepNodes = ['ns_1@172.23.104.213','ns_1@172.23.104.215',

'ns_1@172.23.104.227','ns_1@172.23.105.237',

'ns_1@172.23.105.238','ns_1@172.23.105.63',

'ns_1@172.23.106.109','ns_1@172.23.106.110',

'ns_1@172.23.106.121','ns_1@172.23.106.124',

'ns_1@172.23.106.164','ns_1@172.23.120.167',

'ns_1@172.23.120.59','ns_1@172.23.121.61',

'ns_1@172.23.121.72','ns_1@172.23.121.87',

'ns_1@172.23.121.94','ns_1@172.23.124.27',

'ns_1@172.23.96.170','ns_1@172.23.96.186',

'ns_1@172.23.96.203','ns_1@172.23.96.251',

'ns_1@172.23.96.252','ns_1@172.23.96.253',

'ns_1@172.23.97.189','ns_1@172.23.97.226',

'ns_1@172.23.97.229','ns_1@172.23.97.242',

'ns_1@172.23.97.243','ns_1@172.23.97.244',

'ns_1@172.23.97.245'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 5434f73cc5cd212ee993bcf5fce5cef4

Analytics Service unable to successfully rebalance 7abd22cdd2556f30ca4e515af3920dcf due to 'java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active (missing nodes: [630a974c612881498bf4ec1950b67842], state: ACTIVE)'; see analytics_info.log for details

Rebalance exited with reason {service_rebalance_failed,cbas,

{worker_died,

{'EXIT',<0.14764.2192>,

{task_failed,rebalance,

{service_error,

<<"Rebalance 7abd22cdd2556f30ca4e515af3920dcf failed: timed out waiting for all nodes to join & cluster active (missing nodes: [172.23.121.72:8091 (630a974c612881498bf4ec1950b67842)], state: ACTIVE)">>}}}}}.

Rebalance Operation Id = 5434f73cc5cd212ee993bcf5fce5cef4