Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.2.0
-
7.1.4-3601 --> 7.2.0-5309
-
Untriaged
-
Centos 64-bit
-
0
-
No
Description
Steps to Repro
1. Run a longevity test on 7.1.4 for 4 days.
./sequoia -client 172.23.104.27:2375 -provider file:centos_pine.yml -test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_neo_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.1.4-3601 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. Start an online upgrade using swap rebalance. It failed with MB-56539.
Removed few 7.1.4 nodes and tried rebalance again.
172.23.105.168 10:40:43 PM 19 Apr, 2023
Starting rebalance, KeepNodes = ['ns_1@172.23.104.137','ns_1@172.23.104.155',
|
'ns_1@172.23.104.67','ns_1@172.23.104.69',
|
'ns_1@172.23.104.70','ns_1@172.23.105.107',
|
'ns_1@172.23.105.168','ns_1@172.23.106.100',
|
'ns_1@172.23.106.188','ns_1@172.23.107.131',
|
'ns_1@172.23.107.95','ns_1@172.23.108.103',
|
'ns_1@172.23.120.107','ns_1@172.23.120.245',
|
'ns_1@172.23.121.117','ns_1@172.23.121.86',
|
'ns_1@172.23.123.28','ns_1@172.23.96.148',
|
'ns_1@172.23.96.252','ns_1@172.23.97.119',
|
'ns_1@172.23.97.121','ns_1@172.23.97.122',
|
'ns_1@172.23.97.239','ns_1@172.23.99.20',
|
'ns_1@172.23.99.21'], EjectNodes = ['ns_1@172.23.104.157',
|
'ns_1@172.23.99.25',
|
'ns_1@172.23.96.253',
|
'ns_1@172.23.105.111',
|
'ns_1@172.23.99.11'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = ad50856d59febf00ac4e4ff21d562b4e
|
172.23.106.188 3:21:37 PM 21 Apr, 2023
Analytics Service unable to successfully rebalance 0684fff61e2caca3fa7d5188a3412901 due to 'HYR0114: Node (6736a7344ec35099a2a0aca7bdf0ecb4) is not active'; see analytics_info.log for details
|
172.23.105.168 3:21:38 PM 21 Apr, 2023
Rebalance exited with reason {service_rebalance_failed,cbas,
|
{worker_died,
|
{'EXIT',<0.32038.2704>,
|
{rebalance_failed,
|
{service_error,
|
<<"Rebalance 0684fff61e2caca3fa7d5188a3412901 failed: HYR0114: Node (172.23.99.11:8091 (6736a7344ec35099a2a0aca7bdf0ecb4)) is not active">>}}}}}.
|
Rebalance Operation Id = ad50856d59febf00ac4e4ff21d562b4e
|
cbcollect_info attached.
Attachments
Issue Links
- duplicates
-
MB-56539 [System test upgrade] Online upgrade fails in analytics with "HYR0003: Failure on node 172.23.99.11:8091"
- Closed