Details
Description
Setup:
1.Setup a 18 node cluster. Enable Auto-failover
2.Load data on all 3 buckets [around 50M, 22M, 500k] items.
3. Continue loading data..
4. Remove orchestrator node [105] Add new node [126]
5. Issue rebalance on this cluster. Rebalance failed with "replicator died" - filed bug 5343
6. Re-issue rebalance . Rebalance fails with " change_filter_failed"
Output:
Reissue rebalance fails with "change_filter_failed"
Node 105 is auto-failed over
Rebalance exited with reason {{change_filter_failed,
{'EXIT',
{{badmatch,
{failed,
{error,
{badmatch,
{memcached_error,auth_error,
<<"Auth failure">>}},
[
,
{ebucketmigrator_srv,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}}},
[
,
{misc,'-executing_on_new_process/1-fun-0-', 3}]}}},
[
,
{ns_vbm_sup,'-set_replicas/3-fun-2-',5},
{lists,foldl,3},
{ns_vbm_sup,set_replicas,3},
{ns_vbm_sup,'-set_replicas_on_nodes/3-fun-1-', 3},
{lists,foreach,2},
{ns_vbm_sup,apply_changes,2},
{ns_vbucket_mover,sync_replicas,0}]}
Logs at - https://s3.amazonaws.com/bugdb/jira/bug4-rebalance-181/bug5.tar
Attachments
For Gerrit Dashboard: MB-5594 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
17796,1 | MB-5594 mc_client_binary counterpart of change_vb_filter command. | master | ns_server | Status: MERGED | +2 | +1 |
17803,3 | MB-5594 Use native vbucket filter change when possible. | master | ns_server | Status: MERGED | +2 | +1 |