Details
-
Bug
-
Resolution: Known Error
-
Critical
-
None
-
7.6.0
-
7.6.0-1440
-
Untriaged
-
Centos 64-bit
-
-
0
-
No
Description
Steps:
- 5 node KV cluster with n2n encryption level=all
- Create one magma bucket
- Set auto-fo timeout=1
- Stop memcached on node '172.23.110.65'
- Recover the error again
Observation:
Bucket stuck in warmup on all 5 nodes with memcahced warning messages,
2023-09-14T21:00:27.779570-07:00 WARNING 5278: Invalid password specified for [<ud>@ns_server</ud>]. Mechanism:[PLAIN], UUID:[b2f10202-9332-41c8-bfb7-6e8fd48be97c]
|
2023-09-14T21:00:28.805587-07:00 WARNING 5278: Invalid password specified for [<ud>@ns_server</ud>]. Mechanism:[PLAIN], UUID:[b56c8404-fb8c-4051-a108-e8728fc1def8]
|
Seeing rebalance failure as well after this with log,
=========================CRASH REPORT=========================
|
crasher:
|
initial call: ns_janitor_server:run_cleanup/2
|
pid: <0.2457.130>
|
registered_name: cleanup_process
|
exception error: no match of right hand side value
|
{error,{failed_nodes,['ns_1@172.23.110.64']}}
|
in function ns_janitor:cleanup_apply_config_body/4 (src/ns_janitor.erl, line 428)
|
in call from ns_janitor:'-cleanup_apply_config_on_buckets/3-fun-1-'/3 (src/ns_janitor.erl, line 383)
|
in call from async:'-async_init/4-fun-1-'/3 (src/async.erl, line 199)
|
ancestors: [ns_janitor_server,ns_orchestrator_child_sup,
|
ns_orchestrator_sup,mb_master_sup,mb_master,
|
leader_registry_sup,leader_services_sup,<0.742.0>,
|
ns_server_sup,ns_server_nodes_sup,<0.291.0>,
|
ns_server_cluster_sup,root_sup,<0.155.0>]
|
message_queue_len: 0
|
messages: []
|
links: [<0.27114.1>]
|
dictionary: []
|
trap_exit: false
|
status: running
|
heap_size: 17731
|
stack_size: 28
|
reductions: 2337
|
neighbours:
|
|
|
[error_logger:error,2023-09-14T21:06:25.958-07:00,ns_1@172.23.110.64:logger_proxy<0.71.0>:ale_error_logger_handler:do_log:101]Error in process <0.1535.130> on node 'ns_1@172.23.110.64' with exit value:
|
{{badmatch,{error,{failed_nodes,['ns_1@172.23.110.64']}}},
|
[{ns_janitor,cleanup_apply_config_body,4,
|
[{file,"src/ns_janitor.erl"},{line,428}]},
|
{ns_janitor,'-cleanup_apply_config_on_buckets/3-fun-1-',3,
|
[{file,"src/ns_janitor.erl"},{line,383}]},
|
{async,'-async_init/4-fun-1-',3,[{file,"src/async.erl"},{line,199}]}]}
|
TAF test:
failover.concurrent_failovers.ConcurrentFailoverTests:
|
test_concurrent_failover,nodes_init=5,services_init=kv-kv-kv-kv-kv,replicas=3,maxCount=1,timeout=1,failover_order=kv,failover_method=stop_memcached,bucket_spec=single_bucket.default
|