Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Centos 7 64 bit; Couchbase EE 7.0.0-4342
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Summary:
Seeing a couple of add node failures due to 500 HTTP status errors, both on functional tests jobs and volume test.
During rebalance-in-out
2021-01-30 03:59:13,589 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_cluster', u'type': u'critical', u'node': u'ns_1@172.23.105.175', u'tstamp': 1612007953478L, u'shortText': u'message', u'serverTime': u'2021-01-30T03:59:13.478Z', u'text': u'Add transaction of \'ns_1@172.23.121.74\' failed because of {error,\n complete_join,\n <<"Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\"">>,\n {error,rest_error,\n <<"Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\"">>,\n {bad_status,500,\n <<"[\\"Unexpected server error, request logged.\\"]">>}}}'} |
on .175 (master node), debug.log
[ns_server:warn,2021-01-30T03:59:11.016-08:00,ns_1@172.23.105.175:<0.22034.28>:leader_lease_acquire_worker:handle_exception:250]Failed to acquire lease from 'ns_1@172.23.121.74': {exit, |
{noproc,
|
{gen_server,call,
|
[{leader_lease_agent,
|
'ns_1@172.23.121.74'}, |
{acquire_lease,
|
'ns_1@172.23.105.175', |
<<"d43e0ce4f089b1aff4e4f4932e1e0f55">>, |
[{timeout,15000}, |
{period,15000}]}, |
infinity]}}}
|
[chronicle:warn,2021-01-30T03:59:10.677-08:00,ns_1@172.23.105.175:chronicle_proposer<0.3160.0>:chronicle_proposer:log_busy_peers:1729]Didn't send append request to some peers due to distribution connection being busy. |
Peers:
|
['ns_1@172.23.121.74'] |
and on .74, error.log
[ns_server:error,2021-01-30T03:59:13.465-08:00,ns_1@172.23.121.74:<0.13728.6>:menelaus_util:reply_server_error:203]Server error during processing: ["web request failed", |
{path,"/completeJoin"}, |
{method,'POST'}, |
{type,exit},
|
{what,
|
{{{badmatch,
|
{error,
|
{shutdown,
|
{failed_to_start_child,
|
ns_ssl_services_sup,
|
{shutdown,
|
{failed_to_start_child,
|
ns_rest_ssl_service,
|
{shutdown,
|
{failed_to_start_child,
|
menelaus_web_ipv4,
|
{'EXIT', |
{badarg,
|
[{ets,lookup,[kv,table],[]},
|
{chronicle_kv,get_kv_table,1, |
[{file,"src/chronicle_kv.erl"}, |
{line,825}]}, |
{chronicle_kv,get_fast_path,2, |
[{file,"src/chronicle_kv.erl"}, |
{line,352}]}, |
{chronicle_kv,'-get/3-fun-0-', |
2, |
[{file,"src/chronicle_kv.erl"}, |
{line,329}]}, |
{chronicle_kv,
|
optimistic_query,5, |
[{file,"src/chronicle_kv.erl"}, |
{line,562}]}, |
{chronicle_compat,get,3, |
[{file,
|
Logs attached.
