Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4584
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
In the test,
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
- Continuously run queries on the indexes of collections of bucket1 and bucket2
- wait for 15 mins
- kill cbft on 172.23.97.217 and wait for 15 mins
- stop all mutations and wait for 10 mins
- add fts node 172.23.107.4 and start rebalance and wait for 15 mins
Rebalance failed with below:
Node: 172.23.97.215
[ns_server:error,2021-03-02T16:42:39.810-08:00,ns_1@172.23.97.215:service_rebalancer-fts<0.29586.24>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.29632.24>, |
{rebalance_failed,
|
{service_error,
|
<<"assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex missing, indexDef.Name: bucket_bucket1_idx_flwzs-country-free_parking, pindex: bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2, node: 9a3dcd6b2f5475afe210e3dff6a075c3, state: \"primary\", op: promote, planPIndex: &cbgt.PlanPIndex{Name:\"bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2\", UUID:\"1996c7d3d1ba7b1c\", IndexType:\"fulltext-index\", IndexName:\"bucket_bucket1_idx_flwzs-country-free_parking\", IndexUUID:\"116820e26376e9ab\", IndexParams:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_6.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket1\", SourceUUID:\"26aed6012454cfb3f6fb4c16787a7119\", SourceParams:\"{}\", SourcePartitions:\"728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779\", Nodes:map[string]*cbgt.PlanPIndexNode{\"9fb0278bc883795e26424e7f61c3c899\":(*cbgt.PlanPIndexNode)(0xc0128d7f80), \"c0d85b494365ea8ca151ff7ee70a3c54\":(*cbgt.PlanPIndexNode)(0xc0128d7f90)}}">>}}} |
[user:error,2021-03-02T16:42:39.814-08:00,ns_1@172.23.97.215:<0.8696.0>:ns_orchestrator:log_rebalance_completion:1407]Rebalance exited with reason {service_rebalance_failed,fts, |
{worker_died,
|
{'EXIT',<0.29632.24>, |
{rebalance_failed,
|
{service_error,
|
<<"assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex missing, indexDef.Name: bucket_bucket1_idx_flwzs-country-free_parking, pindex: bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2, node: 9a3dcd6b2f5475afe210e3dff6a075c3, state: \"primary\", op: promote, planPIndex: &cbgt.PlanPIndex{Name:\"bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2\", UUID:\"1996c7d3d1ba7b1c\", IndexType:\"fulltext-index\", IndexName:\"bucket_bucket1_idx_flwzs-country-free_parking\", IndexUUID:\"116820e26376e9ab\", IndexParams:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_6.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket1\", SourceUUID:\"26aed6012454cfb3f6fb4c16787a7119\", SourceParams:\"{}\", SourcePartitions:\"728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779\", Nodes:map[string]*cbgt.PlanPIndexNode{\"9fb0278bc883795e26424e7f61c3c899\":(*cbgt.PlanPIndexNode)(0xc0128d7f80), \"c0d85b494365ea8ca151ff7ee70a3c54\":(*cbgt.PlanPIndexNode)(0xc0128d7f90)}}">>}}}}}. |
Rebalance Operation Id = f460daac5c922252e81232e854c9cf81
|
[ns_server:error,2021-03-02T17:23:26.024-08:00,ns_1@172.23.97.215:<0.27650.42>:diag_handler:handle_diag_eval:785]WARNING: /diag/eval: |
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.237.zip
Attachments
Issue Links
- relates to
-
MB-44725 [System Test]service_rebalance_failed,fts: rebalance_failed,inactivity_timeout
- Closed