Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
None
-
7.1.0
-
Enterprise Edition 7.1.0 build 1312
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Test -
-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
|
Day - 1
Cycle - 1
Scale - 3
Test step:
[2021-09-20T03:51:59-07:00, sequoiatools/couchbase-cli:7.0:255613] server-add -c 172.23.97.74:8091 --server-add https://172.23.120.58 -u Administrator -p password --server-add-username Administrator --server-add-password password --services eventing |
[2021-09-20T03:52:15-07:00, sequoiatools/couchbase-cli:7.0:0eb1bb] rebalance -c 172.23.97.74:8091 --server-remove 172.23.123.33 -u Administrator -p password |
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.23.97.74:8091 --server-remove 172.23.123.33 -u Administrator -p password] |
|
|
docker logs 0eb1bb
|
docker start 0eb1bb
|
|
|
sWARNING: couchbase-cli version 7.0.2-6611-enterprise does not match couchbase server version 7.1.0-1312-enterprise |
*Unable to display progress bar on this os |
JERROR: Rebalance failed. See logs for detailed reason. You can try again. |
On 172.23.97.74
ns_server.debug.log
[ns_server:warn,2021-09-20T05:11:27.098-07:00,ns_1@172.23.97.74:<0.32190.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.123.25'. |
Acquire options were [{timeout,6343},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.095-07:00,ns_1@172.23.97.74:<0.32277.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.96.243'. |
Acquire options were [{timeout,7033},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.442-07:00,ns_1@172.23.97.74:timer_lag_recorder<0.23363.0>:timer_lag_recorder:report_missed_msgs:47]Skipped 8 'check_time' messages |
[ns_server:warn,2021-09-20T05:11:27.564-07:00,ns_1@172.23.97.74:<0.32464.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.112'. |
Acquire options were [{timeout,6355},{period,15000}] |
[chronicle:debug,2021-09-20T05:11:27.099-07:00,ns_1@172.23.97.74:chronicle_leader<0.228.0>:chronicle_leader:handle_state_timeout:571]State timeout when state is: {follower,'ns_1@172.23.97.112', |
<<"745371a8fc247e64030d230081a6cf16">>, |
{5,'ns_1@172.23.97.112'}, |
established}
|
[ns_server:warn,2021-09-20T05:11:27.564-07:00,ns_1@172.23.97.74:<0.32251.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.96.122'. |
Acquire options were [{timeout,6953},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.564-07:00,ns_1@172.23.97.74:<0.10685.103>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.96.14'. |
Acquire options were [{timeout,6963},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.564-07:00,ns_1@172.23.97.74:<0.32076.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.120.73'. |
Acquire options were [{timeout,6962},{period,15000}] |
[ns_server:debug,2021-09-20T05:11:27.565-07:00,ns_1@172.23.97.74:<0.31358.153>:ns_rebalance_observer:docs_left_updater_loop:441]Starting docs_left_updater_loop:"default" |
[]
|
[ns_server:warn,2021-09-20T05:11:27.570-07:00,ns_1@172.23.97.74:<0.32500.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.150'. |
Acquire options were [{timeout,6962},{period,15000}] |
[ns_server:info,2021-09-20T05:11:27.571-07:00,ns_1@172.23.97.74:<0.25330.0>:ns_orchestrator:handle_event:486]Skipping janitor in state rebalancing |
[ns_server:warn,2021-09-20T05:11:27.571-07:00,ns_1@172.23.97.74:<0.32222.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.123.32'. |
Acquire options were [{timeout,6449},{period,15000}] |
[ns_server:debug,2021-09-20T05:11:27.094-07:00,ns_1@172.23.97.74:<0.18993.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 14224790 us |
[ns_server:debug,2021-09-20T05:11:27.094-07:00,ns_1@172.23.97.74:<0.26020.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 14224101 us |
[ns_server:debug,2021-09-20T05:11:27.571-07:00,ns_1@172.23.97.74:<0.20023.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 14701475 us |
[ns_server:warn,2021-09-20T05:11:27.573-07:00,ns_1@172.23.97.74:<0.17159.138>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.96.48'. |
Acquire options were [{timeout,6970},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.573-07:00,ns_1@172.23.97.74:<0.32478.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.148'. |
Acquire options were [{timeout,6342},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.573-07:00,ns_1@172.23.97.74:<0.32048.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.106.134'. |
Acquire options were [{timeout,6348},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.573-07:00,ns_1@172.23.97.74:<0.32377.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.105'. |
Acquire options were [{timeout,6343},{period,15000}] |
[ns_server:debug,2021-09-20T05:11:27.573-07:00,ns_1@172.23.97.74:<0.27875.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 14703414 us |
[ns_server:warn,2021-09-20T05:11:27.574-07:00,ns_1@172.23.97.74:<0.32119.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.120.77'. |
Acquire options were [{timeout,6349},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.574-07:00,ns_1@172.23.97.74:<0.32179.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.123.24'. |
Acquire options were [{timeout,6354},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.574-07:00,ns_1@172.23.97.74:<0.32203.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.123.26'. |
Acquire options were [{timeout,6436},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.574-07:00,ns_1@172.23.97.74:<0.32237.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.123.33'. |
Acquire options were [{timeout,7044},{period,15000}] |
[ns_server:debug,2021-09-20T05:11:27.096-07:00,ns_1@172.23.97.74:<0.23480.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 21611945 us |
[ns_server:warn,2021-09-20T05:11:27.579-07:00,ns_1@172.23.97.74:<0.32514.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.151'. |
Acquire options were [{timeout,6962},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.579-07:00,ns_1@172.23.97.74:<0.32290.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.96.254'. |
Acquire options were [{timeout,6355},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.579-07:00,ns_1@172.23.97.74:<0.32167.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.121.77'. |
Acquire options were [{timeout,7037},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.580-07:00,ns_1@172.23.97.74:<0.32091.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.120.74'. |
Acquire options were [{timeout,6461},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:27.580-07:00,ns_1@172.23.97.74:<0.32486.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.149'. |
Acquire options were [{timeout,7045},{period,15000}] |
[ns_server:debug,2021-09-20T05:11:28.487-07:00,ns_1@172.23.97.74:<0.21573.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 914693 us |
[ns_server:error,2021-09-20T05:11:28.488-07:00,ns_1@172.23.97.74:service_status_keeper_worker<0.23578.0>:service_status_keeper:grab_status:200]Got exception while getting items from node 'ns_1@172.23.106.134': {exit, |
{timeout,
|
{gen_server,
|
call,
|
[{remote_api,
|
'ns_1@172.23.106.134'}, |
get_fts_indexes,
|
10000]}}} |
[ns_server:debug,2021-09-20T05:11:28.489-07:00,ns_1@172.23.97.74:leader_activities<0.25298.0>:leader_activities:terminate_activities:612]Terminating activities (reason is {shutdown, |
{quorum_lost,
|
{lease_lost,'ns_1@172.23.123.26'}}}): |
[{activity,<0.27570.153>,#Ref<0.2001790067.4026007558.71517>,default, |
<<"adc77ca4b3cf278140d85e1fec5f4275">>, |
[rebalance],
|
majority,[]}]
|
[ns_server:debug,2021-09-20T05:11:28.490-07:00,ns_1@172.23.97.74:<0.20022.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 910129 us |
[ns_server:debug,2021-09-20T05:11:28.490-07:00,ns_1@172.23.97.74:<0.16090.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 23272435 us |
[ns_server:debug,2021-09-20T05:11:28.492-07:00,ns_1@172.23.97.74:dcp_traffic_monitor<0.23642.183>:dcp_traffic_monitor:update_bucket:124]Saw that bucket "bucket8" became alive on node 'ns_1@172.23.123.26' |
[ns_server:debug,2021-09-20T05:11:28.492-07:00,ns_1@172.23.97.74:dcp_traffic_monitor<0.23642.183>:dcp_traffic_monitor:update_bucket:124]Saw that bucket "bucket8" became alive on node 'ns_1@172.23.123.25' |
[ns_server:debug,2021-09-20T05:11:28.492-07:00,ns_1@172.23.97.74:dcp_traffic_monitor<0.23642.183>:dcp_traffic_monitor:update_bucket:124]Saw that bucket "bucket9" became alive on node 'ns_1@172.23.120.77' |
[ns_server:error,2021-09-20T05:11:28.498-07:00,ns_1@172.23.97.74:service_status_keeper-fts<0.23584.0>:service_status_keeper:handle_cast:103]Service service_fts returned incorrect status |
[ns_server:debug,2021-09-20T05:11:28.499-07:00,ns_1@172.23.97.74:<0.29314.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 928026 us |
[ns_server:debug,2021-09-20T05:11:28.501-07:00,ns_1@172.23.97.74:<0.23481.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 930225 us |
[ns_server:warn,2021-09-20T05:11:28.501-07:00,ns_1@172.23.97.74:<0.32105.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.120.75'. |
Acquire options were [{timeout,177},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:28.501-07:00,ns_1@172.23.97.74:<0.32527.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.97.241'. |
Acquire options were [{timeout,845},{period,15000}] |
[ns_server:warn,2021-09-20T05:11:28.502-07:00,ns_1@172.23.97.74:<0.29126.153>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.120.58'. |
Acquire options were [{timeout,847},{period,15000}] |
[error_logger:error,2021-09-20T05:11:28.483-07:00,ns_1@172.23.97.74:node_status_analyzer<0.22211.183>:ale_error_logger_handler:do_log:101] |
=========================ERROR REPORT=========================
|
** Generic server node_status_analyzer terminating
|
** Last message in was refresh
|
** When Server state == {state,
|
{dict,29,16,16,8,80,48, |
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
|
{{[['ns_1@172.23.97.110'| |
{healthy,-576442677566414545}], |
['ns_1@172.23.97.150'| |
{healthy,-576442677566347112}]], |
[['ns_1@172.23.120.81'| |
{healthy,-576442677571113813}], |
['ns_1@172.23.123.31'| |
{healthy,-576442677567694966}], |
['ns_1@172.23.97.151'| |
{healthy,-576442677566330551}], |
['ns_1@172.23.97.241'| |
{healthy,-576442677566115121}]], |
[['ns_1@172.23.123.32'| |
{healthy,-576442677567679154}], |
['ns_1@172.23.96.122'| |
{healthy,-576442677567391000}], |
['ns_1@172.23.97.112'| |
{healthy,-576442677566396528}]], |
[['ns_1@172.23.120.73'| |
{healthy,-576442677571655068}], |
['ns_1@172.23.123.33'| |
{healthy,-576442677567664224}], |
['ns_1@172.23.96.243'| |
{healthy,-576442677566480555}]], |
[['ns_1@172.23.106.134'| |
{healthy,-576442677572832736}], |
['ns_1@172.23.120.74'| |
{healthy,-576442677571630021}], |
['ns_1@172.23.123.24'| |
{healthy,-576442677568511045}], |
['ns_1@172.23.96.14'| |
{healthy,-576442677566506973}], |
['ns_1@172.23.96.254'| |
{healthy,-576442677566463259}], |
['ns_1@172.23.97.74'| |
{healthy,-576442676555939866}]], |
[['ns_1@172.23.120.75'| |
{healthy,-576442677571616468}], |
['ns_1@172.23.123.25'| |
{healthy,-576442677568269915}], |
['ns_1@172.23.97.105'| |
{healthy,-576442677566430825}]], |
[['ns_1@172.23.120.86'| |
{healthy,-576442677570333880}], |
['ns_1@172.23.123.26'| |
{healthy,-576442677567719515}]], |
[['ns_1@172.23.120.77'| |
{healthy,-576442677571155130}], |
['ns_1@172.23.121.77'| |
{healthy,-576442677569750208}]], |
[['ns_1@172.23.120.58'| |
{healthy,-576442677572818768}], |
['ns_1@172.23.96.48'| |
{healthy,-576442677566446661}], |
['ns_1@172.23.97.148'| |
{healthy,-576442677566380147}]], |
[['ns_1@172.23.97.149'| |
{healthy,-576442677566364627}]], |
[],[],[],[],[],[]}}},
|
['ns_1@172.23.106.134','ns_1@172.23.120.58', |
'ns_1@172.23.120.73','ns_1@172.23.120.74', |
'ns_1@172.23.120.75','ns_1@172.23.120.77', |
'ns_1@172.23.120.81','ns_1@172.23.120.86', |
'ns_1@172.23.121.77','ns_1@172.23.123.24', |
'ns_1@172.23.123.25','ns_1@172.23.123.26', |
'ns_1@172.23.123.31','ns_1@172.23.123.32', |
'ns_1@172.23.123.33','ns_1@172.23.96.122', |
'ns_1@172.23.96.14','ns_1@172.23.96.243', |
'ns_1@172.23.96.254','ns_1@172.23.96.48', |
'ns_1@172.23.97.105','ns_1@172.23.97.110', |
'ns_1@172.23.97.112','ns_1@172.23.97.148', |
'ns_1@172.23.97.149','ns_1@172.23.97.150', |
'ns_1@172.23.97.151','ns_1@172.23.97.241', |
'ns_1@172.23.97.74'], |
node_status_analyzer}
|
** Reason for termination == |
** {{timeout,{gen_server,call,[node_monitor,get_nodes]}},
|
[{gen_server,call,2,[{file,"gen_server.erl"},{line,215}]}, |
{node_status_analyzer,handle_info,3, |
[{file,"src/node_status_analyzer.erl"},{line,84}]}, |
{health_monitor,handle_message,3, |
[{file,"src/health_monitor.erl"},{line,100}]}, |
{health_monitor,handle_info,2, |
[{file,"src/health_monitor.erl"},{line,85}]}, |
{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]}, |
{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,711}]}, |
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} |
|
|
[ns_server:debug,2021-09-20T05:11:27.565-07:00,ns_1@172.23.97.74:<0.16092.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 14697468 us |
[ns_server:warn,2021-09-20T05:11:28.540-07:00,ns_1@172.23.97.74:<0.32147.9>:leader_lease_acquire_worker:handle_acquire_timeout:106]Timeout while trying to acquire lease from 'ns_1@172.23.120.86'. |
Acquire options were [{timeout,843},{period,15000}] |
[ns_server:debug,2021-09-20T05:11:28.536-07:00,ns_1@172.23.97.74:dcp_traffic_monitor<0.23642.183>:dcp_traffic_monitor:update_bucket:124]Saw that bucket "ITEM" became alive on node 'ns_1@172.23.120.73' |
[ns_server:debug,2021-09-20T05:11:28.542-07:00,ns_1@172.23.97.74:<0.14909.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 968478 us |
[ns_server:debug,2021-09-20T05:11:28.500-07:00,ns_1@172.23.97.74:<0.27878.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 926458 us |
[ns_server:debug,2021-09-20T05:11:28.565-07:00,ns_1@172.23.97.74:<0.23562.183>:ns_memcached:perform_very_long_call_with_timing:849]ensure_bucket took too long: 12886482 us |
[ns_server:debug,2021-09-20T05:11:28.539-07:00,ns_1@172.23.97.74:<0.22119.153>:ns_rebalancer:rebalance_topology_aware_service:414]Got an exit signal while waiting for the service rebalance to complete. Service: index. Exit message: {'EXIT', |
<0.27570.153>, |
shutdown}
|
[ns_server:debug,2021-09-20T05:11:28.579-07:00,ns_1@172.23.97.74:<0.21575.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 15708924 us |
[ns_server:error,2021-09-20T05:11:28.579-07:00,ns_1@172.23.97.74:service_rebalancer-index<0.4168.154>:service_rebalancer:run_rebalance_worker:122]Got exit message from parent: {'EXIT',<0.22119.153>,shutdown} |
[ns_server:warn,2021-09-20T05:11:28.535-07:00,ns_1@172.23.97.74:goxdcr_status_keeper<0.23570.0>:goxdcr_status_keeper:refresh:73]Failed to refresh goxdcr replications: {exit, |
{timeout,
|
{gen_server,call,[cb_dist,status]}},
|
[{gen_server,call,2, |
[{file,"gen_server.erl"}, |
{line,215}]}, |
{cb_dist,get_config,0, |
[{file,"src/cb_dist.erl"}, |
{line,226}]}, |
{cb_dist,address_family,0, |
[{file,"src/cb_dist.erl"}, |
{line,216}]}, |
{misc,localhost,1, |
[{file,"src/misc.erl"},{line,1611}]}, |
{misc,local_url,3, |
[{file,"src/misc.erl"},{line,1660}]}, |
{goxdcr_rest,send_with_timeout,5, |
[{file,"src/goxdcr_rest.erl"}, |
{line,52}]}, |
{goxdcr_rest,query_goxdcr,4, |
[{file,"src/goxdcr_rest.erl"}, |
{line,112}]}, |
{goxdcr_rest,get_from_goxdcr,3, |
[{file,"src/goxdcr_rest.erl"}, |
{line,135}]}]} |
[ns_server:info,2021-09-20T05:11:28.583-07:00,ns_1@172.23.97.74:rebalance_agent<0.23679.0>:rebalance_agent:handle_down:290]Rebalancer process <0.22119.153> died (reason shutdown). |
[ns_server:warn,2021-09-20T05:11:28.580-07:00,ns_1@172.23.97.74:goxdcr_status_keeper<0.23570.0>:goxdcr_status_keeper:handle_info:58]Ignoring unknown msg: {#Ref<0.2001790067.4026793987.218923>, |
[{name,ns_1},
|
{config,
|
[{external_listeners,[inet_tls_dist]},
|
{local_listeners,[inet_tcp_dist]},
|
{preferred_external_proto,inet_tls_dist},
|
{preferred_local_proto,inet_tcp_dist}]},
|
{listeners,
|
[{{local,inet_tcp_dist},
|
{#Port<0.4972>, |
{net_address,
|
{{127,0,0,1},21100}, |
"localhost",tcp,inet}, |
0}}, |
{{external,inet_tls_dist},
|
{#Port<0.4771>, |
{net_address,
|
{{0,0,0,0},21150}, |
"localhost",tls,inet}, |
0}}]}, |
{acceptors,
|
[{<0.32758.9>,{local,inet_tcp_dist}}, |
{<0.20463.9>,{external,inet_tls_dist}}]}, |
{connections,
|
[{con,#Ref<0.2001790067.4025745413.205826>, |
inet_tls_dist,<0.29486.153>, |
#Ref<0.2001790067.4025745413.205828>}, |
{con,#Ref<0.2001790067.4024958983.230406>, |
inet_tls_dist,<0.17354.138>, |
#Ref<0.2001790067.4024958983.230407>}, |
{con,#Ref<0.2001790067.4023648261.4077>, |
inet_tls_dist,<0.10622.103>, |
#Ref<0.2001790067.4023648260.31205>}, |
{con,#Ref<0.2001790067.4019453953.188221>, |
inet_tls_dist,<0.32517.9>, |
#Ref<0.2001790067.4019453953.188223>}, |
{con,#Ref<0.2001790067.4019453960.98610>, |
inet_tls_dist,<0.32503.9>, |
#Ref<0.2001790067.4019453960.98612>}, |
{con,#Ref<0.2001790067.4019453960.98564>, |
inet_tls_dist,<0.32489.9>, |
#Ref<0.2001790067.4019453960.98566>}, |
{con,#Ref<0.2001790067.4019453959.116360>, |
inet_tls_dist,<0.32481.9>, |
#Ref<0.2001790067.4019453959.116362>}, |
{con,#Ref<0.2001790067.4019453957.139822>, |
inet_tls_dist,<0.32467.9>, |
#Ref<0.2001790067.4019453955.151052>}, |
{con,#Ref<0.2001790067.4019453955.151042>, |
inet_tls_dist,<0.32395.9>, |
#Ref<0.2001790067.4019453955.151044>}, |
{con,#Ref<0.2001790067.4019453956.165894>, |
inet_tls_dist,<0.32381.9>, |
#Ref<0.2001790067.4019453953.188111>}, |
{con,#Ref<0.2001790067.4019453954.176191>, |
inet_tls_dist,<0.32303.9>, |
#Ref<0.2001790067.4019453954.176192>}, |
{con,#Ref<0.2001790067.4019453960.98212>, |
inet_tls_dist,<0.32280.9>, |
#Ref<0.2001790067.4019453956.165747>}, |
{con,#Ref<0.2001790067.4019453953.187803>, |
inet_tls_dist,<0.32266.9>, |
#Ref<0.2001790067.4019453953.187805>}, |
{con,#Ref<0.2001790067.4019453954.176010>, |
inet_tls_dist,<0.32240.9>, |
#Ref<0.2001790067.4019453954.176012>}, |
{con,#Ref<0.2001790067.4019453956.165645>, |
inet_tls_dist,<0.32224.9>, |
#Ref<0.2001790067.4019453959.115878>}, |
{con,#Ref<0.2001790067.4019453957.139422>, |
inet_tls_dist,<0.32218.9>, |
#Ref<0.2001790067.4019453957.139424>}, |
{con,#Ref<0.2001790067.4019453956.165604>, |
inet_tls_dist,<0.32206.9>, |
#Ref<0.2001790067.4019453956.165605>}, |
{con,#Ref<0.2001790067.4019453953.187546>, |
inet_tls_dist,<0.32194.9>, |
#Ref<0.2001790067.4019453953.187548>}, |
{con,#Ref<0.2001790067.4019453955.150743>, |
inet_tls_dist,<0.32182.9>, |
#Ref<0.2001790067.4019453954.175809>}, |
{con,#Ref<0.2001790067.4019453958.168399>, |
inet_tls_dist,<0.32170.9>, |
#Ref<0.2001790067.4019453958.168400>}, |
{con,#Ref<0.2001790067.4019453954.175759>, |
inet_tls_dist,<0.32150.9>, |
#Ref<0.2001790067.4019453954.175761>}, |
{con,#Ref<0.2001790067.4019453957.139243>, |
inet_tls_dist,<0.32138.9>, |
#Ref<0.2001790067.4019453957.139245>}, |
{con,#Ref<0.2001790067.4019453960.97894>, |
inet_tls_dist,<0.32122.9>, |
#Ref<0.2001790067.4019453960.97896>}, |
{con,#Ref<0.2001790067.4019453960.97850>, |
inet_tls_dist,<0.32108.9>, |
#Ref<0.2001790067.4019453960.97851>}, |
{con,#Ref<0.2001790067.4019453960.97825>, |
inet_tls_dist,<0.32094.9>, |
#Ref<0.2001790067.4019453960.97826>}, |
{con,#Ref<0.2001790067.4019453955.150395>, |
inet_tls_dist,<0.32079.9>, |
#Ref<0.2001790067.4019453955.150397>}, |
{con,#Ref<0.2001790067.4019453958.168134>, |
inet_tls_dist,<0.32063.9>, |
#Ref<0.2001790067.4019453958.168136>}, |
{con,#Ref<0.2001790067.4019453957.139019>, |
inet_tls_dist,<0.32034.9>, |
#Ref<0.2001790067.4019453957.139020>}, |
{con,#Ref<0.2001790067.4019191814.71659>, |
inet_tcp_dist,<0.25182.0>, |
#Ref<0.2001790067.4019191814.71661>}, |
{con,#Ref<0.2001790067.4019191811.69431>, |
inet_tcp_dist,<0.25170.0>, |
#Ref<0.2001790067.4019191811.69433>}]}]} |
[ns_server:debug,2021-09-20T05:11:28.585-07:00,ns_1@172.23.97.74:<0.26017.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 1004869 us |
[ns_server:debug,2021-09-20T05:11:28.585-07:00,ns_1@172.23.97.74:<0.18991.1>:ns_memcached:verify_report_long_call:344]call verify_warmup took too long: 1006048 us |
[ns_server:error,2021-09-20T05:11:28.592-07:00,ns_1@172.23.97.74:<0.31605.153>:leader_activities:report_error:992]Activity {default,rebalance} failed with error {quorum_lost, |
{lease_lost,
|
'ns_1@172.23.123.26'}} |
[error_logger:error,2021-09-20T05:11:28.593-07:00,ns_1@172.23.97.74:<0.31468.153>:ale_error_logger_handler:do_log:101] |
=========================CRASH REPORT=========================
|
crasher:
|
initial call: erlang:apply/2 |
pid: <0.31468.153> |
registered_name: []
|
exception error: no match of right hand side value
|
{leader_activities_error,
|
{default,rebalance}, |
{quorum_lost,{lease_lost,'ns_1@172.23.123.26'}}} |
in function ns_rebalancer:rebalance/5 (src/ns_rebalancer.erl, line 484) |
ancestors: [<0.25330.0>,ns_orchestrator_child_sup,ns_orchestrator_sup, |
mb_master_sup,mb_master,leader_registry_sup,
|
leader_services_sup,<0.23614.0>,ns_server_sup, |
ns_server_nodes_sup,<0.260.0>,ns_server_cluster_sup, |
root_sup,<0.140.0>] |
message_queue_len: 0 |
messages: []
|
links: [<0.25330.0>] |
dictionary: []
|
trap_exit: false |
status: running
|
heap_size: 75113 |
stack_size: 27 |
reductions: 12523 |
neighbours:
|
|
|
[user:error,2021-09-20T05:11:28.593-07:00,ns_1@172.23.97.74:<0.25330.0>:ns_orchestrator:log_rebalance_completion:1412]Rebalance exited with reason {{badmatch, |
{leader_activities_error,
|
{default,rebalance}, |
{quorum_lost,
|
{lease_lost,'ns_1@172.23.123.26'}}}}, |
[{ns_rebalancer,rebalance,5, |
[{file,"src/ns_rebalancer.erl"},{line,484}]}, |
{proc_lib,init_p_do_apply,3, |
[{file,"proc_lib.erl"},{line,249}]}]}. |
Rebalance Operation Id = 276f68e48d4e51ea4829a948d2879d7d
|