Details
Major Description
Test that led to failure:
1. Created a 3 node cluster
2. Created a bucket and loaded 3M items.
3. Started a swap rebalance by adding a new node and starting rebalance by ejecting out an existing node.
4. Rotated internal user password on the master node by calling the API /node/controller/rotateInternalCredentials. Got the expected error
["System is being reconfigured. Please try later."] |
5. But the rebalance operation failed with the following error
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.21208.47>, |
{{badarg,
|
[{erlang,link,
|
[undefined],
|
[{error_info,
|
#{module => erl_erts_errors}}]},
|
{janitor_agent,'-do_handle_call/3-fun-5-', |
3, |
[{file,"src/janitor_agent.erl"}, |
{line,750}]}, |
{janitor_agent,
|
'-spawn_rebalance_subprocess/3-fun-0-',3, |
[{file,"src/janitor_agent.erl"}, |
{line,947}]}, |
{proc_lib,init_p,3, |
[{file,"proc_lib.erl"},{line,225}]}]}, |
{gen_server,call,
|
[{'janitor_agent-bucket-0', |
'ns_1@172.23.105.208'}, |
{if_rebalance,<0.18799.47>, |
{update_vbucket_state,1009,active, |
undefined,undefined,
|
[['ns_1@172.23.105.208', |
'ns_1@172.23.105.206', |
'ns_1@172.23.105.211']]}}, |
infinity]}}}}}.
|
Rebalance Operation Id = 27c2270c9395563f14d2dd8538619d74
|
TAF script to reproduce:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini rerun=False,get-cbcollect-info=False,skip_cluster_reset=True,upgrade_version=7.6.0-1813 -t security.internal_password.InternalUserPassword.test_password_rotation_rebalance,nodes_init=3,rebalance_type=swap_rebalance,nodes_in=1,nodes_out=1' |
TAF ref job: debian-security_security-password-rotation