Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
Cheshire-Cat
-
1
Description
Was: [System Test] : Rebalance failing because of reason - badmatch - leader_activities_error - no_quorum
Build : 7.0.0-4735
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 2
Iteration : 1st
In the longevity tests, 12 rebalance operations have failed because of the following error. The following stack trace is from the first rebalance failure.
[ns_server:error,2021-03-21T12:53:37.641-07:00,ns_1@172.23.97.74:<0.7176.1045>:leader_activities:report_error:998]Activity {default,{ns_janitor,"ITEM",apply_config}} failed with error {no_quorum,
|
[{required_quorum,
|
{all,
|
{set,
|
9,
|
16,
|
16,
|
8,
|
80,
|
48,
|
{[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[]},
|
{{[],
|
['ns_1@172.23.97.241'],
|
['ns_1@172.23.97.112'],
|
['ns_1@172.23.120.73'],
|
['ns_1@172.23.97.74',
|
'ns_1@172.23.123.24'],
|
['ns_1@172.23.123.25'],
|
['ns_1@172.23.123.26',
|
'ns_1@172.23.120.86'],
|
['ns_1@172.23.120.77'],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[],
|
[]}}}}},
|
{leases,
|
['ns_1@172.23.97.110',
|
'ns_1@172.23.97.150',
|
'ns_1@172.23.123.31',
|
'ns_1@172.23.97.151',
|
'ns_1@172.23.120.81',
|
'ns_1@172.23.123.32',
|
'ns_1@172.23.97.112',
|
'ns_1@172.23.120.73',
|
'ns_1@172.23.123.33',
|
'ns_1@172.23.97.74',
|
'ns_1@172.23.106.134',
|
'ns_1@172.23.120.74',
|
'ns_1@172.23.123.24',
|
'ns_1@172.23.96.254',
|
'ns_1@172.23.123.25',
|
'ns_1@172.23.120.75',
|
'ns_1@172.23.123.26',
|
'ns_1@172.23.106.136',
|
'ns_1@172.23.120.86',
|
'ns_1@172.23.120.77',
|
'ns_1@172.23.97.148',
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-21T12:53:37.644-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
[{required_quorum,
|
{all,
|
{set,9,16,16,8,80,48,
|
{[],[],[],[],[],[],[],[],[],[],[],[],[],
|
[],[],[]},
|
{{[],
|
['ns_1@172.23.97.241'],
|
['ns_1@172.23.97.112'],
|
['ns_1@172.23.120.73'],
|
['ns_1@172.23.97.74',
|
'ns_1@172.23.123.24'],
|
['ns_1@172.23.123.25'],
|
['ns_1@172.23.123.26',
|
'ns_1@172.23.120.86'],
|
['ns_1@172.23.120.77'],
|
[],[],[],[],[],[],[],[]}}}}},
|
{leases,
|
['ns_1@172.23.97.110','ns_1@172.23.97.150',
|
'ns_1@172.23.123.31','ns_1@172.23.97.151',
|
'ns_1@172.23.120.81','ns_1@172.23.123.32',
|
'ns_1@172.23.97.112','ns_1@172.23.120.73',
|
'ns_1@172.23.123.33','ns_1@172.23.97.74',
|
'ns_1@172.23.106.134',
|
'ns_1@172.23.120.74','ns_1@172.23.123.24',
|
'ns_1@172.23.96.254','ns_1@172.23.123.25',
|
'ns_1@172.23.120.75','ns_1@172.23.123.26',
|
'ns_1@172.23.106.136',
|
'ns_1@172.23.120.86','ns_1@172.23.120.77',
|
'ns_1@172.23.97.148','ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}}},
|
[{ns_janitor,cleanup_apply_config,4,
|
[{file,"src/ns_janitor.erl"},{line,215}]},
|
{ns_janitor,
|
'-cleanup_membase_bucket/4-fun-0-',4,
|
[{file,"src/ns_janitor.erl"},{line,66}]},
|
{async,'-async_init/4-fun-1-',3,
|
[{file,"src/async.erl"},{line,197}]}]}.
|
Rebalance Operation Id = 4ed109d13d9f0bb4908809d95ceab1bf
|
The other rebalance failures are -
[user:error,2021-03-21T12:53:37.644-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-21T13:07:53.954-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-21T18:51:36.410-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-21T21:57:18.172-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T00:53:16.143-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T02:00:41.284-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T02:25:59.065-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T02:33:50.240-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T02:43:30.808-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T02:56:48.088-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T02:59:38.757-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|
--
|
'ns_1@172.23.96.48',
|
'ns_1@172.23.120.58',
|
'ns_1@172.23.97.149']}]}
|
[user:error,2021-03-22T03:03:53.473-07:00,ns_1@172.23.97.74:<0.23085.0>:ns_orchestrator:log_rebalance_completion:1410]Rebalance exited with reason {{badmatch,
|
{leader_activities_error,
|
{default,{ns_janitor,"ITEM",apply_config}},
|
{no_quorum,
|