Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
7.6.0
-
Couchbase server
Enterprise Edition 7.6.0 build 2119
-
Untriaged
-
0
-
Yes
Description
1. Create 6 node cluster. (cluster_services ::['kv', 'kv,fts', 'kv,fts', 'kv', 'fts', 'fts'])
2. Create a bucket 'standard_bucket_1'. Load 10000 docs into the bucket(non vector data).
3. Create a second bucket 'vector-bucket-0' and load siftsmall dataset which has 10k docs.
4. Create index 'standard_bucket_1_index_1' on 'stand_bucket_1'.
5. Create vector index 'vector_vector-bucket-0_index_0' on 'vector-bucket-0'.
6. Wait for indexing to complete
7. Add a new node to the cluster and start a rebalance in operation
{'knownNodes': 'ns_1@172.23.109.165,ns_1@172.23.122.236,ns_1@172.23.216.218,ns_1@172.23.217.110,ns_1@172.23.217.79,ns_1@172.23.217.81,ns_1@172.23.218.9', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} |
8. While rebalance was in progress ran queries on the indexes. Example of a vector query
{'query': {'match_none': {}}, 'explain': True, 'fields': ['*'], 'knn': [{'field': 'vector_data', 'k': 100, 'vector': [1.0, 3.0, 11.0, 110.0, 62.0, 22.0, 4.0, 0.0, 43.0, 21.0, 22.0, 18.0, 6.0, 28.0, 64.0, 9.0, 11.0, 1.0, 0.0, 0.0, 1.0, 40.0, 101.0, 21.0, 20.0, 2.0, 4.0, 2.0, 2.0, 9.0, 18.0, 35.0, 1.0, 1.0, 7.0, 25.0, 108.0, 116.0, 63.0, 2.0, 0.0, 0.0, 11.0, 74.0, 40.0, 101.0, 116.0, 3.0, 33.0, 1.0, 1.0, 11.0, 14.0, 18.0, 116.0, 116.0, 68.0, 12.0, 5.0, 4.0, 2.0, 2.0, 9.0, 102.0, 17.0, 3.0, 10.0, 18.0, 8.0, 15.0, 67.0, 63.0, 15.0, 0.0, 14.0, 116.0, 80.0, 0.0, 2.0, 22.0, 96.0, 37.0, 28.0, 88.0, 43.0, 1.0, 4.0, 18.0, 116.0, 51.0, 5.0, 11.0, 32.0, 14.0, 8.0, 23.0, 44.0, 17.0, 12.0, 9.0, 0.0, 0.0, 19.0, 37.0, 85.0, 18.0, 16.0, 104.0, 22.0, 6.0, 2.0, 26.0, 12.0, 58.0, 67.0, 82.0, 25.0, 12.0, 2.0, 2.0, 25.0, 18.0, 8.0, 2.0, 19.0, 42.0, 48.0, 11.0]}]} |
9. Vector queries start to fail.
10. In logs I see a lot of these error messages
2024-02-11T07:34:14.016-08:00 [WARN] grpc_client: Query() returned error from host: 172.23.109.165:9130, err: grpc_client: query got status code: 500, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc018d1e1a0), Request:(*bleve.SearchRequest)(0xc0186460f0), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = Unknown desc = query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:1532024-02-11T07:34:14.370-08:00 [WARN] grpc_client: Query() returned error from host: 172.23.109.165:9130, err: grpc_client: query got status code: 500, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc018d1e580), Request:(*bleve.SearchRequest)(0xc00064e3c0), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = Unknown desc = query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:1532024-02-11T07:34:14.598-08:00 [WARN] grpc_client: Query() returned error from host: 172.23.109.165:9130, err: grpc_client: query got status code: 500, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc018d1e980), Request:(*bleve.SearchRequest)(0xc00064e4b0), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = Unknown desc = query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:1532024-02-11T07:34:14.680-08:00 [WARN] grpc_client: Query() returned error from host: 172.23.109.165:9130, err: grpc_client: query got status code: 500, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc01947f9a0), Request:(*bleve.SearchRequest)(0xc0002e8870), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = Unknown desc = query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:1532024-02-11T07:34:15.396-08:00 [WARN] grpc_client: Query() returned error from host: 172.23.109.165:9130, err: grpc_client: query got status code: 500, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc018d1ed80), Request:(*bleve.SearchRequest)(0xc0186461e0), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = Unknown desc = query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:153 |
Marking this as blocker since all regression tests are failing as every query is failing in the tests: http://qa.sc.couchbase.com/job/test_suite_executor/678003/consoleFull
