Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
Untriaged
-
0
-
Unknown
Description
Seems like we are hitting rpc limit of 50MB when query is trying to return more than 50MB. When we say we support 10000 results documents for higher dimensions, it may not work. This may be expected error given that rpc limit, but it either needs be documented or fixed to return the results in pagination manner.
Ex:
- Vector size = 1536 * 4Bytes = 6144Bytes
- Other text fields may have 2000 Bytes extra, total document size may be around 8K
- For 10000, it goes beyond 80MB which is the limit we are hitting.
So in my case 7000 Documents are amounting up to 62549551 bytes which is greater than 50MB.
Example Document:
"c1": "c1", "dim": 1536, "id": "vect109000", "s1": "s1", "sname": "fefg", "sno": 108999, "vector_data": [1, 3, 4, ....] |
Cluster Config:
- 5 Dedicated FTS nodes with 16vCPUs and 64GiB RAM
- Number of documents indexed by FTS 42974894
- Dimension: 1536
Error message:
grpc_client: query got status code: 429, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc0b5694100), Request:(*bleve.SearchRequest)(0xc000d7cc30), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = ResourceExhausted desc = trying to send message larger than max (62549551 vs. 52428800) |
|
FTS Query:
{ "query":{ "match_none":{ } }, "explain":true, "knn":[{ "field":"vector_data", "k":7000, "vector":[116,34,46,3,1,6,27,44,0,9,77,32,37,83,41,3,19,14,15,17,18,34,22,22,12,23,52,12,0,0,0,1,8,22,15,16,16,35,92,94,1,3,9,13,22,120,120,12,120,15,5,0,0,39,74,120,115,24,14,7,2,7,4,34,1,18,36,120,41,10,2,0,1,8,93,120,35,21,7,0,120,104,57,7,0,1,2,20,120,51,2,13,18,12,3,20,7,6,9,18,18,24,19,4,5,12,114,71,13,2,0,2,60,30,81,54,0,0,0,13,44,10,1,3,7,15,20,38,26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0, 26, 22, 8, 3, 7, 5, 13, 23, 24, 4, 1, 13, 47, 57, 93, 49, 109, 63, 61, 15, 17, 17, 46, 109, 1, 6, 86, 104, 19, 2, 40, 45, 9, 7, 12, 12, 34, 29, 12, 4, 8, 9, 7, 26, 44, 61, 35, 12, 47, 53, 34, 3, 3, 5, 86, 109, 109, 31, 23, 21, 10, 2, 67, 109, 16, 15, 7, 9, 23, 29, 22, 17, 14, 51, 38, 53, 44, 20, 6, 13, 76, 109, 78, 6, 0, 1, 1, 21, 90, 54, 60, 4, 2, 20, 8, 34, 22, 51, 43, 10, 7, 5, 14, 8, 31, 109, 45, 36, 28, 11, 3, 43, 23, 109, 109, 14, 10, 11, 10, 35, 11, 17, 22, 6, 35, 105, 8, 0] }],"ctl": {"timeout": 300000, "consistency": {"level": "at_plus"}} } |
Attachments
| For Gerrit Dashboard: MB-60844 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 207416,5 | MB-60844, MB-61216: Interpret empty `query` (in search request) as match_none | trinity | cbft | Status: MERGED | +2 | +1 |
| 207510,3 | MB-60844, MB-61216: go mod tidy | trinity | cbftx | Status: MERGED | +2 | +1 |
| 208179,1 | Merge remote-tracking branch 'couchbase/trinity' | master | cbft | Status: MERGED | +2 | +1 |