Details
-
Bug
-
Resolution: Incomplete
-
Major
-
None
-
7.1.0
-
Untriaged
-
1
-
Unknown
-
KV 2021-Nov, KV 2021-Dec
Description
Build : 7.1.0-1624
Test Script & Scope : -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml
Setup:
- Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
- Create 1 bucket, 100 scopes and 10 collections in each scopes
- Create 2500 GSI indexes ( 5 on each collection)
- Load documents on some of the collections
- Created 750 indexes: one index (1 partition) on each collection
failover/addback/rebalance of the node ( 172.31.43.62) failed with "ERROR: Internal server error, please retry your request". In the logs we see below
[ns_server:info,2021-11-12T10:35:49.585Z,ns_1@172.31.39.188:<0.1799.2039>:ns_janitor:cleanup_with_membase_bucket_vbucket_map:126]Bucket "bucket1" not yet ready on ['ns_1@172.31.39.188']
|
[ns_server:info,2021-11-12T10:35:49.585Z,ns_1@172.31.39.188:rebalance_agent<0.765.0>:rebalance_agent:handle_down:290]Rebalancer process <0.871.2040> died (reason {pre_rebalance_janitor_run_failed,
|
"bucket1",
|
{error,
|
wait_for_memcached_failed,
|
['ns_1@172.31.39.188']}}).
|
[ns_server:debug,2021-11-12T10:35:49.585Z,ns_1@172.31.39.188:leader_activities<0.21112.1>:leader_activities:handle_activity_down:505]Activity terminated with reason {shutdown,
|
{async_died,
|
{raised,
|
{exit,
|
{pre_rebalance_janitor_run_failed,
|
"bucket1",
|
{error,wait_for_memcached_failed,
|
['ns_1@172.31.39.188']}},
|
[{ns_rebalancer,
|
run_janitor_pre_rebalance,1,
|
[{file,"src/ns_rebalancer.erl"},
|
{line,643}]},
|
{lists,foreach,2,
|
[{file,"lists.erl"},{line,1342}]},
|
{ns_rebalancer,rebalance_body,5,
|
[{file,"src/ns_rebalancer.erl"},
|
{line,478}]},
|
{async,'-async_init/4-fun-1-',3,
|
[{file,"src/async.erl"},
|
{line,191}]}]}}}}. Activity:
|
|
|
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636715163/collectinfo-2021-11-12T110604-ns_1%40172.31.39.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636715163/collectinfo-2021-11-12T110604-ns_1%40172.31.40.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636715163/collectinfo-2021-11-12T110604-ns_1%40172.31.43.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1636715163/collectinfo-2021-11-12T110604-ns_1%40172.31.43.62.zip