Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4766
Componet Clusterops test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
Test with index create/drop loop.
Test steps:
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
- Continuously run queries on the indexes of collections of bucket1 and bucket2
- wait for 15 mins
- kill cbft on 172.23.97.217 and wait for 15 mins
- add fts node 172.23.107.4 and start rebalance and wait for 15 mins
Rebalance runs for 14hrs and failed with below
2021-03-24T16:40:11.815-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.215) - Rebalance exited with reason {service_rebalance_failed,fts,
|
{worker_died,
|
{'EXIT',<0.14892.26>,
|
{rebalance_failed,
|
{service_error,
|
<<"assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex already exists, indexDef: &cbgt.IndexDef{Type:\"fulltext-index\", Name:\"bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92\", UUID:\"7724d08ef29d4d0e\", Params:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_4.coll_0\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_2\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_3\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_4\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket4\", SourceUUID:\"4ce7bb91b974daa8a2b85f7b5c69bfef\", SourceParams:\"null\", PlanParams:cbgt.PlanParams{MaxPartitionsPerPIndex:69, IndexPartitions:15, NumReplicas:2, HierarchyRules:blance.HierarchyRules(nil), NodePlanParams:map[string]map[string]*cbgt.NodePlanParam(nil), PIndexWeights:map[string]int(nil), PlanFrozen:false}}, pindex: bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92_7724d08ef29d4d0e_c386b63c, node: f3f85682f2eecd2f8205805942ba59af, state: \"replica\", op: add, planPIndex: &cbgt.PlanPIndex{Name:\"bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92_7724d08ef29d4d0e_c386b63c\", UUID:\"47b27a09cdb0136b\", IndexType:\"fulltext-index\", IndexName:\"bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92\", IndexUUID:\"7724d08ef29d4d0e\", IndexParams:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_4.coll_0\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_2\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_3\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_4\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket4\", SourceUUID:\"4ce7bb91b974daa8a2b85f7b5c69bfef\", SourceParams:\"{}\", SourcePartitions:\"207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275\", Nodes:map[string]*cbgt.PlanPIndexNode{\"3378566cf2f2f37381f4e38c4856e492\":(*cbgt.PlanPIndexNode)(0xc1e9bcc4f0), \"f3f85682f2eecd2f8205805942ba59af\":(*cbgt.PlanPIndexNode)(0xc1e9bcc500)}}">>}}}}}.
|
Rebalance Operation Id = cce9e24a91a2353caaa9ecc89b5d5454
|
cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616643988/collectinfo-2021-03-25T034631-ns_1%40172.23.97.236.zip
########## Cluster config ##################
|
###### fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091] ########### |
###### kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091] ########### |
###### n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091] ########### |