Steps to Repo
1. Start a 1 node KV cluster
2. Add 83 more nodes (11 KV nodes and each other service with 12 nodes. So there would be 12 backup service nodes in the cluster). Rebalance the cluster. This rebalance fails.
Rebalance exited with reason {service_rebalance_failed,backup,
|
{agent_died,<33228.15506.2>,
|
{linked_process_died,<33228.18655.2>,
|
{'ns_1@172.23.122.151',
|
{{badmatch,
|
{false,
|
{topology,[],[],false,[]},
|
{topology,[],
|
[<<"25f9131e242cb6fd9c4dd86f740b7240">>,
|
<<"37a97e7ada233fd4a849a41171b8b8af">>,
|
<<"396db5abf9128578b050cfd169529e74">>,
|
<<"7fa75f59dcc1603850349f5e51c60862">>,
|
<<"9a2a8208209c21ab2587b5b66a42d814">>,
|
<<"a377bb65e29e451e5786eac463d7dbfc">>,
|
<<"d91fde258c5436e53209526da73375a5">>],
|
true,[]}}},
|
[{service_agent,long_poll_worker_loop,5,
|
[{file,"src/service_agent.erl"},
|
{line,605}]},
|
{proc_lib,init_p,3,
|
[{file,"proc_lib.erl"},{line,211}]}]}}}}}.
|
Rebalance Operation Id = 525b55e1f43e7dacde960d57b2967c13
|
Observations
On ns_1@172.23.122.151:
Service 'backup' exited with status 1. Restarting. Messages:
|
2022-01-24T10:52:01.328-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools'
|
2022-01-24T10:52:01.328-08:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default'
|
2022-01-24T10:52:01.762-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default'
|
2022-01-24T10:52:01.768-08:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/buckets'
|
2022-01-24T10:52:01.807-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/buckets'
|
2022-01-24T10:52:01.808-08:00 INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"1ebbb73ab21b19610609379830b21647","version":{"min_version":"7.1.0","is_mixed_cluster":false},"max_vbuckets":0,"uniform_vbuckets":true}
|
2022-01-24T10:52:01.808-08:00 INFO (Node) Confirmed cluster uuid {"clusterUUID": "1ebbb73ab21b19610609379830b21647"}
|
2022-01-24T10:52:02.121-08:00 ERROR (Main) Failed to run node {"err": "could not get service configuration: Rev mismatch"}
|
Bug
Critical