Loading...

Details

Type: Bug
Resolution: Not a Bug
Priority: Critical
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: fts
Labels:
- eagle-eye
- system-test

Triage:
Untriaged
Story Points:
1
Is this a Regression?:
Unknown

Description

Build: 7.0.0-4766
Componet Clusterops test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1

Test with index create/drop loop.

Test steps:

there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
Continuously run queries on the indexes of collections of bucket1 and bucket2
wait for 15 mins
kill cbft on 172.23.97.217 and wait for 15 mins
add fts node 172.23.107.4 and start rebalance and wait for 15 mins

Rebalance runs for 14hrs and failed with below

2021-03-24T16:40:11.815-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.97.215) - Rebalance exited with reason {service_rebalance_failed,fts,

                              {worker_died,

                               {'EXIT',<0.14892.26>,

                                {rebalance_failed,

                                 {service_error,

                                  <<"assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex already exists, indexDef: &cbgt.IndexDef{Type:\"fulltext-index\", Name:\"bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92\", UUID:\"7724d08ef29d4d0e\", Params:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_4.coll_0\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_2\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_3\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_4\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket4\", SourceUUID:\"4ce7bb91b974daa8a2b85f7b5c69bfef\", SourceParams:\"null\", PlanParams:cbgt.PlanParams{MaxPartitionsPerPIndex:69, IndexPartitions:15, NumReplicas:2, HierarchyRules:blance.HierarchyRules(nil), NodePlanParams:map[string]map[string]*cbgt.NodePlanParam(nil), PIndexWeights:map[string]int(nil), PlanFrozen:false}}, pindex: bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92_7724d08ef29d4d0e_c386b63c, node: f3f85682f2eecd2f8205805942ba59af, state: \"replica\", op: add, planPIndex: &cbgt.PlanPIndex{Name:\"bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92_7724d08ef29d4d0e_c386b63c\", UUID:\"47b27a09cdb0136b\", IndexType:\"fulltext-index\", IndexName:\"bucket_bucket4_idx_checb-type-country-public_likes-free_parking-Overall_92\", IndexUUID:\"7724d08ef29d4d0e\", IndexParams:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_4.coll_0\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_2\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_3\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}},\\\"scope_4.coll_4\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]},\\\"public_likes\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"public_likes\\\",\\\"type\\\":\\\"text\\\"}]},\\\"reviews\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"date\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"date\\\",\\\"type\\\":\\\"datetime\\\"}]},\\\"ratings\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"Overall\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"Overall\\\",\\\"type\\\":\\\"number\\\"}]}}}}}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket4\", SourceUUID:\"4ce7bb91b974daa8a2b85f7b5c69bfef\", SourceParams:\"{}\", SourcePartitions:\"207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275\", Nodes:map[string]*cbgt.PlanPIndexNode{\"3378566cf2f2f37381f4e38c4856e492\":(*cbgt.PlanPIndexNode)(0xc1e9bcc4f0), \"f3f85682f2eecd2f8205805942ba59af\":(*cbgt.PlanPIndexNode)(0xc1e9bcc500)}}">>}}}}}.

Rebalance Operation Id = cce9e24a91a2353caaa9ecc89b5d5454

cbcollect logs:

########## Cluster config ##################

######  fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]  ###########

######  kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]  ###########

######  n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]  ###########

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[System Test] service_rebalance_failed,fts:worker_died:service_error

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty