Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4766
Componet Clusterops test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
Test with index create/drop loop.
Test steps:
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
- Continuously run queries on the indexes of collections of bucket1 and bucket2
- wait for 15 mins
- kill cbft on 172.23.97.217 and wait for 15 mins
- add fts node 172.23.107.4 and start rebalance and wait for 15 mins
Seeing lot of "[ERRO] rest: error code: 400, msg: rest_create_index: index type is required" thoughout the test.
172.23.107.4 : fts
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-03-24T12:26:07.282-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-03-24T12:28:31.900-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-03-24T12:28:34.701-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-03-24T12:29:36.865-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-03-24T12:08:32.935-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-03-24T12:08:41.180-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-03-24T12:20:45.002-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-03-24T12:24:14.873-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-03-24T12:24:27.800-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:38:36.166-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:39:20.952-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:40:13.109-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:42:00.489-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:46:35.995-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:47:08.804-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:47:33.555-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:48:36.951-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:52:08.543-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T11:53:26.281-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.2.gz:2021-03-24T12:04:19.192-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.3.gz:2021-03-24T11:20:18.596-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.3.gz:2021-03-24T11:20:40.997-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.3.gz:2021-03-24T11:36:29.614-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.3.gz:2021-03-24T11:37:00.970-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.3.gz:2021-03-24T11:37:14.155-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.4.gz:2021-03-24T11:08:57.192-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.4.gz:2021-03-24T11:09:14.731-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.4.gz:2021-03-24T11:11:10.767-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required -- rest.ShowErrorBody() at rest.go:63
|
As part of the test, we are setting the indexType to scorch in the create index request. From the logs, not able to relate to any create request sent by the test.
For example:
172.23.107.4 : fts
/opt/couchbase/var/lib/couchbase/logs/fts.log:2021-03-24T12:26:07.282-07:00 [ERRO] rest: error code: 400, msg: rest_create_index: index type is required – rest.ShowErrorBody() at rest.go:63
At this point there were 2 index create threads were running and none of them failed to create index.
Thread 1:
{"log":"2021-03-24 05:24:41,421 - ftsindexmanager - INFO - ===== Creating single FTS index on ['scope_2.coll_8'] =====\n","stream":"stderr","time":"2021-03-24T12:24:41.421990568Z"}
|
{"log":"2021-03-24 05:24:53,870 - ftsindexmanager - INFO - FTS indexes in cluster - : \n","stream":"stderr","time":"2021-03-24T12:24:53.871337547Z"}
|
{"log":"['bucket_bucket1_idx_agajc-price', 'bucket_bucket1_idx_evzvn-type-Overall-price-free_parking', 'bucket_bucket1_idx_fhllw-country-price-type-public_likes-Overall-free_parking', 'bucket_bucket1_idx_hcqfg-Overall-country-public_likes-price-type-free_parking', 'bucket_bucket1_idx_hgzkq-country-price-Overall-type-free_parking-public_likes', 'bucket_bucket1_idx_ihjac-type-price', 'bucket_bucket1_idx_ldygi-free_parking-type-country-price-public_likes', 'bucket_bucket1_idx_locux-price-public_likes-country-Overall-free_parking-type', 'bucket_bucket1_idx_qftoa-country-public_likes-Overall', 'bucket_bucket1_idx_snrai-price-country-free_parking-public_likes', 'bucket_bucket1_idx_vurql-Overall-free_parking-country-price-public_likes', 'bucket_bucket1_idx_wmgfk-type', 'bucket_bucket2_idx_cxbkr-type-public_likes-price-Overall-country-free_parking', 'bucket_bucket2_idx_enkit-country-free_parking-public_likes', 'bucket_bucket2_idx_hhetg-country-price-type', 'bucket_bucket2_idx_ivadl-free_parking-price-type-public_likes-country-Overall', 'bucket_bucket2_idx_muevu-type-public_likes', 'bucket_bucket2_idx_ncxws-country-price-public_likes-type-Overall', 'bucket_bucket2_idx_nqizz-Overall-free_parking', 'bucket_bucket2_idx_pqlrt-country-public_likes', 'bucket_bucket2_idx_qrnkb-type-free_parking', 'bucket_bucket2_idx_qrnpf-price-country-free_parking-type-Overall-public_likes', 'bucket_bucket3_idx_dtkot-country-public_likes', 'bucket_bucket3_idx_epkeo-public_likes-type', 'bucket_bucket3_idx_fqfih-price-country-public_likes-type-Overall-free_parking', 'bucket_bucket3_idx_iagxc-type-free_parking-price-public_likes-country-Overall', 'bucket_bucket3_idx_mnyux-public_likes-country', 'bucket_bucket3_idx_mpdlm-country-price-public_likes-type-free_parking-Overall', 'bucket_bucket3_idx_nawrg-type-price', 'bucket_bucket3_idx_nhuht-Overall-price-country', 'bucket_bucket3_idx_ploqv-Overall-free_parking-public_likes-country-price', 'bucket_bucket3_idx_qykji-type', 'bucket_bucket3_idx_rsyse-free_parking-price-public_likes-country-Overall-type', 'bucket_bucket3_idx_tnhec-public_likes-type-free_parking-Overall-price', 'bucket_bucket3_idx_unyut-Overall-price-free_parking-type-country', 'bucket_bucket3_idx_vhboi-public_likes', 'bucket_bucket3_idx_vxxns-price', 'bucket_bucket3_idx_vzbbv-type-country-public_likes', 'bucket_bucket3_idx_xphby-public_likes-type-free_parking-Overall-country-price', 'bucket_bucket3_idx_ygwzr-type-public_likes-price', 'bucket_bucket4_idx_oswmz-Overall-type-country-price-public_likes_33', 'bucket_bucket5_idx_eknqp-price_34']\n","stream":"stderr","time":"2021-03-24T12:24:53.871368814Z"}
|
{"log":"2021-03-24 05:24:53,871 - ftsindexmanager - INFO - Creating FTS index bucket_bucket5_idx_vrzlt-Overall-price-free_parking_36 on ['scope_2.coll_8'] with 1 replicas and 6 partitions\n","stream":"stderr","time":"2021-03-24T12:24:53.872371205Z"}
|
{"log":"2021-03-24 05:25:20,451 - ftsindexmanager - INFO - Index creation on bucket_bucket5_idx_vrzlt-Overall-price-free_parking_36 succeeded. Content = {'status': 'ok', 'uuid': '60566f4891a62c69'}, Response = {'content-type': 'application/json;version=2.0.0', 'date': 'Wed, 24 Mar 2021 12:25:20 GMT', 'content-length': '42', 'status': '200'}\n","stream":"stderr","time":"2021-03-24T12:25:20.452139423Z"}
|
{"log":"2021-03-24 05:29:20,552 - ftsindexmanager - INFO - Deleting index bucket_bucket5_idx_vrzlt-Overall-price-free_parking_36\n","stream":"stderr","time":"2021-03-24T12:29:20.552509058Z"}
|
Thread 2:
{"log":"2021-03-24 05:25:10,326 - ftsindexmanager - INFO - ===== Creating single FTS index on ['scope_0.coll_2'] =====\n","stream":"stderr","time":"2021-03-24T12:25:10.327020604Z"}
|
{"log":"2021-03-24 05:26:09,697 - ftsindexmanager - INFO - FTS indexes in cluster - : \n","stream":"stderr","time":"2021-03-24T12:26:09.698115615Z"}
|
{"log":"['bucket_bucket1_idx_agajc-price', 'bucket_bucket1_idx_evzvn-type-Overall-price-free_parking', 'bucket_bucket1_idx_fhllw-country-price-type-public_likes-Overall-free_parking', 'bucket_bucket1_idx_hcqfg-Overall-country-public_likes-price-type-free_parking', 'bucket_bucket1_idx_hgzkq-country-price-Overall-type-free_parking-public_likes', 'bucket_bucket1_idx_ihjac-type-price', 'bucket_bucket1_idx_ldygi-free_parking-type-country-price-public_likes', 'bucket_bucket1_idx_locux-price-public_likes-country-Overall-free_parking-type', 'bucket_bucket1_idx_qftoa-country-public_likes-Overall', 'bucket_bucket1_idx_snrai-price-country-free_parking-public_likes', 'bucket_bucket1_idx_vurql-Overall-free_parking-country-price-public_likes', 'bucket_bucket1_idx_wmgfk-type', 'bucket_bucket2_idx_cxbkr-type-public_likes-price-Overall-country-free_parking', 'bucket_bucket2_idx_enkit-country-free_parking-public_likes', 'bucket_bucket2_idx_hhetg-country-price-type', 'bucket_bucket2_idx_ivadl-free_parking-price-type-public_likes-country-Overall', 'bucket_bucket2_idx_muevu-type-public_likes', 'bucket_bucket2_idx_ncxws-country-price-public_likes-type-Overall', 'bucket_bucket2_idx_nqizz-Overall-free_parking', 'bucket_bucket2_idx_pqlrt-country-public_likes', 'bucket_bucket2_idx_qrnkb-type-free_parking', 'bucket_bucket2_idx_qrnpf-price-country-free_parking-type-Overall-public_likes', 'bucket_bucket3_idx_dtkot-country-public_likes', 'bucket_bucket3_idx_epkeo-public_likes-type', 'bucket_bucket3_idx_fqfih-price-country-public_likes-type-Overall-free_parking', 'bucket_bucket3_idx_iagxc-type-free_parking-price-public_likes-country-Overall', 'bucket_bucket3_idx_mnyux-public_likes-country', 'bucket_bucket3_idx_mpdlm-country-price-public_likes-type-free_parking-Overall', 'bucket_bucket3_idx_nawrg-type-price', 'bucket_bucket3_idx_nhuht-Overall-price-country', 'bucket_bucket3_idx_ploqv-Overall-free_parking-public_likes-country-price', 'bucket_bucket3_idx_qykji-type', 'bucket_bucket3_idx_rsyse-free_parking-price-public_likes-country-Overall-type', 'bucket_bucket3_idx_tnhec-public_likes-type-free_parking-Overall-price', 'bucket_bucket3_idx_unyut-Overall-price-free_parking-type-country', 'bucket_bucket3_idx_vhboi-public_likes', 'bucket_bucket3_idx_vxxns-price', 'bucket_bucket3_idx_vzbbv-type-country-public_likes', 'bucket_bucket3_idx_xphby-public_likes-type-free_parking-Overall-country-price', 'bucket_bucket3_idx_ygwzr-type-public_likes-price', 'bucket_bucket4_idx_oswmz-Overall-type-country-price-public_likes_33', 'bucket_bucket5_idx_eknqp-price_34']\n","stream":"stderr","time":"2021-03-24T12:26:09.698148959Z"}
|
{"log":"2021-03-24 05:26:09,698 - ftsindexmanager - INFO - Creating FTS index bucket_bucket4_idx_ktyen-country-Overall-public_likes-type-price_37 on ['scope_0.coll_2'] with 0 replicas and 20 partitions\n","stream":"stderr","time":"2021-03-24T12:26:09.6984492Z"}
|
{"log":"2021-03-24 05:26:09,698 - ftsindexmanager - INFO - Field reviews already exists\n","stream":"stderr","time":"2021-03-24T12:26:09.698578476Z"}
|
{"log":"2021-03-24 05:26:29,478 - ftsindexmanager - INFO - Index creation on bucket_bucket4_idx_ktyen-country-Overall-public_likes-type-price_37 succeeded. Content = {'status': 'ok', 'uuid': '2e9f9acff576619a'}, Response = {'content-type': 'application/json;version=2.0.0', 'date': 'Wed, 24 Mar 2021 12:26:29 GMT', 'content-length': '42', 'status': '200'}\n","stream":"stderr","time":"2021-03-24T12:26:29.478725971Z"}
|
{"log":"2021-03-24 05:30:29,484 - ftsindexmanager - INFO - Deleting index bucket_bucket4_idx_ktyen-country-Overall-public_likes-type-price_37\n","stream":"stderr","time":"2021-03-24T12:30:29.484718148Z"}
|
cbcollect logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1616615137/collectinfo-2021-03-24T194539-ns_1%40172.23.97.237.zip
########## Cluster config ##################
|
###### fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091] ########### |
###### kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091] ########### |
###### n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091] ########### |
Attachments
Issue Links
- duplicates
-
MB-44476 [System Test]cannot create index because an index with the same name already exists
-
- Reopened
-