Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.1.0
-
6.6.5-10076 —> 7.1.0-2117
-
Untriaged
-
Centos 64-bit
-
1
-
No
Description
Steps to Repro
1. Run 6.6.5 longevity test for 5-6 days.
./sequoia -client 172.23.104.254:2375 -provider file:centos_second_cluster.yml -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml -scale 3 -repeat 0 -log_level 0 -version 6.6.5-10076 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
|
2. Online upgrade to 7.1 using swap rebalance and graceful failover/recovery strategies.
Post upgrade of all the 27 nodes saw the following error on the UI.
ns_1@172.23.120.197 3:00:47 AM 22 Jan, 2022
Unsuccessful user storage upgrade. {error, {badmatch, {error, [{'ns_1@172.23.97.149', {exit, {noproc, {gen_server,call, [{ns_config_rep,'ns_1@172.23.97.149'}, synchronize_everything,infinity]}}}}]}}, [{menelaus_users,upgrade,3,[{file,"src/menelaus_users.erl"},{line,914}]}, {cluster_compat_mode,do_upgrades,5, [{file,"src/cluster_compat_mode.erl"},{line,244}]}, {cluster_compat_mode,do_consider_switching_compat_mode,3, [{file,"src/cluster_compat_mode.erl"},{line,266}]}, {compat_mode_manager,handle_consider_switching_compat_mode,0, [{file,"src/compat_mode_manager.erl"},{line,40}]}, {compat_mode_manager,handle_call,3, [{file,"src/compat_mode_manager.erl"},{line,31}]}, {gen_server2,handle_call,3,[{file,"src/gen_server2.erl"},{line,214}]}, {gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,721}]}, {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,750}]}]} (repeated 1 times, last seen 38.652576 secs ago)
|
On 172.23.120.197
[ns_server:error,2022-01-22T03:00:16.983-08:00,ns_1@172.23.120.197:compat_mode_manager<0.12256.0>:ns_config_rep:synchronize_remote:361]Failed to synchronize config to some nodes:
|
[{'ns_1@172.23.97.149',
|
{exit,
|
{noproc,
|
{gen_server,call,
|
[{ns_config_rep,'ns_1@172.23.97.149'},
|
synchronize_everything,infinity]}}}}]
|
[user:error,2022-01-22T03:00:16.983-08:00,ns_1@172.23.120.197:compat_mode_manager<0.12256.0>:cluster_compat_mode:complete_ns_config_upgrade:313]Was unable to sync cluster_compat_version update to some nodes: [{'ns_1@172.23.97.149',
|
{exit,
|
{noproc,
|
{gen_server,
|
call,
|
[{ns_config_rep,
|
'ns_1@172.23.97.149'},
|
synchronize_everything,
|
infinity]}}}}]
|
On 172.23.97.149
[ns_server:error,2022-01-22T03:00:08.882-08:00,ns_1@172.23.97.149:<0.27156.294>:menelaus_util:reply_server_error_before_close:209]Server error during processing: ["web request failed",
|
{path,"/pools/default/bs/default"},
|
{method,'GET'},
|
{type,exit},
|
{what,
|
{noproc,
|
{gen_server,call,
|
[request_tracker,
|
{note_request,<0.27156.294>,rest},
|
infinity]}}},
|
{trace,
|
[{gen_server,call,3,
|
[{file,"gen_server.erl"},{line,247}]},
|
{request_tracker,request,2,
|
[{file,"src/request_tracker.erl"},
|
{line,34}]},
|
{menelaus_util,handle_request,2,
|
[{file,"src/menelaus_util.erl"},
|
{line,220}]},
|
{mochiweb_http,headers,6,
|
[{file,
|
"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
|
{line,150}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,226}]}]}]
|
[ns_server:error,2022-01-22T03:00:14.826-08:00,ns_1@172.23.97.149:<0.29656.294>:menelaus_util:reply_server_error_before_close:209]Server error during processing: ["web request failed",
|
{path,"/_cbauth/checkPermission"},
|
{method,'GET'},
|
{type,exit},
|
{what,
|
{{function_clause,
|
[{menelaus_roles,compile_params,
|
[[],
|
[any,any],
|
[{"default",
|
<<"bd13cf068ae17c679b51df0dd2ced271">>}],
|
undefined],
|
[{file,"src/menelaus_roles.erl"},
|
{line,876}]},
|
{menelaus_roles,compile_role,4,
|
[{file,"src/menelaus_roles.erl"},
|
{line,892}]},
|
{lists,filtermap,2,
|
[{file,"lists.erl"},{line,1321}]},
|
{menelaus_users,'-make_props/4-fun-3-',
|
9,
|
[{file,"src/menelaus_users.erl"},
|
{line,363}]},
|
{menelaus_users,'-make_props/4-fun-4-',
|
3,
|
[{file,"src/menelaus_users.erl"},
|
{line,385}]},
|
{lists,mapfoldl,3,
|
[{file,"lists.erl"},{line,1358}]},
|
{menelaus_users,make_props,4,
|
[{file,"src/menelaus_users.erl"},
|
{line,383}]},
|
{menelaus_users,get_roles,1,
|
[{file,"src/menelaus_users.erl"},
|
{line,617}]}]},
|
{gen_server,call,
|
[compiled_roles_cache,
|
{get_and_cache,{"queryUser4",local}}]}}},
|
{trace,
|
[{gen_server,call,2,
|
[{file,"gen_server.erl"},{line,239}]},
|
{menelaus_roles,is_allowed,2,
|
[{file,"src/menelaus_roles.erl"},
|
{line,784}]},
|
{menelaus_web_rbac,
|
handle_check_permission_for_cbauth,1,
|
[{file,"src/menelaus_web_rbac.erl"},
|
{line,1495}]},
|
{request_tracker,request,2,
|
[{file,"src/request_tracker.erl"},
|
{line,40}]},
|
{menelaus_util,handle_request,2,
|
[{file,"src/menelaus_util.erl"},
|
{line,220}]},
|
{mochiweb_http,headers,6,
|
[{file,
|
"/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},
|
{line,150}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,226}]}]}]
|
|
cbcollect_info attached. This is the first time we are running system test upgrade to 7.1.