Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4733
Componet Clusterops test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
Test steps:
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
- Continuously run queries on the indexes of collections of bucket1 and bucket2
- wait for 15 mins
- kill cbft on 172.23.97.217 and wait for 15 mins
- add fts node 172.23.107.4 and start rebalance and wait for 15 mins
Rebalance ran for ~ 10 hrs and failed with below:
2021-03-19T10:45:39.469-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.215) - Rebalance exited with reason {service_rebalance_failed,fts, |
{worker_died,
|
{'EXIT',<0.6283.26>, |
{rebalance_failed,
|
{service_error,
|
<<"nodes: sample, res: (*http.Response)(nil), urlUUID: monitor.UrlUUID{Url:\"http://172.23.107.4:8094\", UUID:\"966d938419bda73b77b2611b069530af\"}, kind: /api/stats?partitions=true, err: Get http://@fts-cbauth:***@172.23.107.4:8094/api/stats?partitions=true: EOF">>}}}}}. |
Rebalance Operation Id = 388c4f8fc3586427a8f8a59e30fd45a3
|
2021-03-19T10:46:02.460-07:00, ns_log:0:info:message(ns_1@172.23.107.5) - Service 'fts' exited with status 137. Restarting. Messages: |
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616180619/collectinfo-2021-03-19T190342-ns_1%40172.23.97.237.zip
########## Cluster config ##################
|
###### fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091] ########### |
###### kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091] ########### |
###### n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091] ########### |
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-