Details
-
Bug
-
Resolution: Incomplete
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4603
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
In the test,
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
- Continuously run queries on the indexes of collections of bucket1 and bucket2
- wait for 15 mins
- kill cbft on 172.23.97.217 and wait for 15 mins
- stop all mutations and wait for 10 mins
- add fts node 172.23.107.4 and start rebalance and wait for 15 mins
- stop create index loop on bucket4 and bucket5
- wait for rebalance to complete
- Once rebalance is complete, kill cbft on 172.23.107.5 and wait for 15 mins
- start mutations on the collections of bucket1, bucket2 and buckt3
- wait for 2 mins and rebalance to remove node 172.23.97.232 and wait for 5 mins
- kill memcached on 172.23.97.237 and wait for 15 mins
- Add data node 172.23.97.232 and rebalance and wait for 10 mins
Rebalance failed with below:
2021-03-04T01:33:03.419-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.215) - Rebalance exited with reason {mover_crashed, |
{unexpected_exit,
|
{'EXIT',<0.31095.90>, |
{{{{badmatch,{error,einval}},
|
[{ns_memcached,worker_loop,3, |
[{file,"src/ns_memcached.erl"}, |
{line,203}]}, |
{proc_lib,init_p_do_apply,3, |
[{file,"proc_lib.erl"},{line,249}]}]}, |
{gen_server,call,
|
['ns_memcached-bucket3', |
{get_dcp_docs_estimate,636, |
"replication:ns_1@172.23.97.235->ns_1@172.23.97.232:bucket3"}, |
180000]}}, |
{gen_server,call,
|
[{'janitor_agent-bucket3', |
'ns_1@172.23.97.235'}, |
{if_rebalance,<0.12751.72>, |
{wait_dcp_data_move,
|
['ns_1@172.23.97.232'], |
635}}, |
infinity]}}}}}.
|
- rebalance out fts node : 172.23.97.216, which failed with below:
2021-03-04T02:46:47.293-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.215) - Rebalance exited with reason {mover_crashed, |
{unexpected_exit,
|
{'EXIT',<0.10720.96>, |
{{{{nocatch,{error,closed}},
|
[{mc_binary,recv_with_data,4, |
[{file,"src/mc_binary.erl"},{line,47}]}, |
{mc_binary,quick_active_recv,3, |
[{file,"src/mc_binary.erl"},{line,54}]}, |
{mc_binary,quick_stats_loop_enter,5, |
[{file,"src/mc_binary.erl"},{line,106}]}, |
{mc_binary,quick_stats,5, |
[{file,"src/mc_binary.erl"},{line,91}]}, |
{mc_client_binary,get_dcp_docs_estimate,
|
3, |
[{file,"src/mc_client_binary.erl"}, |
{line,781}]}, |
{ns_memcached,do_handle_call,3, |
[{file,"src/ns_memcached.erl"}, |
{line,588}]}, |
{ns_memcached,worker_loop,3, |
[{file,"src/ns_memcached.erl"}, |
{line,224}]}, |
{proc_lib,init_p_do_apply,3, |
[{file,"proc_lib.erl"},{line,249}]}]}, |
{gen_server,call,
|
['ns_memcached-bucket3', |
{get_dcp_docs_estimate,58, |
"replication:ns_1@172.23.97.215->ns_1@172.23.97.232:bucket3"}, |
180000]}}, |
{gen_server,call,
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614887985/collectinfo-2021-03-04T195946-ns_1%40172.23.97.237.zip
