Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
7.2.1
-
Enterprise Edition 7.2.1 build 5866
-
Untriaged
-
0
-
Yes
Description
Not 100% sure on the exact steps to reproduce but this did occur a few times, so presumably should work.
Create a cluster with KV + GSI + Query nodes.
Add a new indexer node and rebalance in.
Go to the newly added node and kill indexer. This kills the rebalance
Now add the node again and rebalance in again. This leads to rebalance failure with this error.
{agent_died,<32687.4448.254>, |
{linked_process_died,<32687.6787.254>, |
{'ns_1@172.23.97.108', |
{timeout,
|
{gen_server,call,
|
[<32687.5557.254>, |
{call,"ServiceAPI.PrepareTopologyChange", |
#Fun<json_rpc_connection.0.69248800>, |
#{timeout => 60000}}, |
60000]}}}}}}. |
Could be due to the recent changes for MB-57057. I've marked this as a test blocker because the system test does a lot of indexer kills.
cbcollect ->
/https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.105.122.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.171.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.176.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.30.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.198.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.230.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.245.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.100.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.108.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.109.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.66.zip |
https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.67.zip |
Attachments
Issue Links
| For Gerrit Dashboard: MB-57907 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 194227,2 | MB-57907 Abort early on rebalance cancel | unstable | indexing | Status: MERGED | +2 | +1 |
| 194228,2 | MB-57907 Fix RState change ordering for proxy instance | unstable | indexing | Status: MERGED | +2 | +1 |
| 194239,1 | MB-57968 [BP 7.2.1] Abort early on rebalance cancel | neo | indexing | Status: MERGED | +2 | +1 |
| 194240,1 | MB-57968 [BP 7.2.1] Fix RState change ordering for proxy instance | neo | indexing | Status: MERGED | +2 | +1 |