Details
Description
Attempted to do the following following steps:
- load 40k items on node1
- add node2
- add node3 (stop rebalance)
- failover and rebalance out node1
-Error while rebalancing out node1
Here's an excerpt from the log and I have archived the cluster_run logs here:
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.4121.0>,
{wait_seqno_persisted_failed,"default",5,5000,
[{'n_2@10.0.0.105',
{'EXIT',
{{{{{badmatch,
[{<19180.3595.0>,
{done,exit,
{{badarg,
[{io_lib,format,
["Unexpected response. Unrecognised opaque ~p (~p, ~p)",
[7,
]],
[
,
{line,133}]},
{'ale_logger-rebalance',warn,5,
[
,
{line,30}]},
{upr_consumer_conn,
process_add_close_stream_response,
3,
[
{upr_consumer_conn,
handle_packet,5,
[{file, "src/upr_consumer_conn.erl"}
,
{line,62}]},{upr_proxy,handle_packet,4,
[{file,"src/upr_proxy.erl"}, {line,114}]},
{mc_replication,process_data,3,
[{file, "src/mc_replication.erl"},{line,62}
]},
{mc_replication,process_data,4,
[
,
{line,79}]},
{upr_proxy,handle_info,2,
[
,
{line,84}]}]},
{gen_server,call,
[<19180.3490.0>,
,
infinity]}},
[{gen_server,call,3,
[
{upr_replicator,
'
[{file,"src/upr_replicator.erl"}, {line,117}]},
{upr_replicator,
'
[{file,"src/upr_replicator.erl"}, {line,173}]}]}}]},
[{misc,
sync_shutdown_many_i_am_trapping_exits,
1,
[{file,"src/misc.erl"},{line,1475}]},
{upr_replicator,spawn_and_wait,1,
[{file,"src/upr_replicator.erl"}, {line,195}]},
{upr_replicator,handle_call,3,
[{file,"src/upr_replicator.erl"}, {line,116}]},
{gen_server,handle_msg,5,
[{file,"gen_server.erl"}
,
{line,588}]},
{proc_lib,init_p_do_apply,3,
[
,
{line,227}]}]},
{gen_server,call,
['upr_replicator-default-n_1@10.0.0.105',
,
infinity]}},
{gen_server,call,
['replication_manager-default',
,
infinity]}},
{gen_server,call,
[
,
{if_rebalance,<0.4034.0>,
{wait_seqno_persisted,5,5000}},
infinity]}}}}]}}}