Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.6.2
-
Enterprise Edition 7.6.2 build 3710
-
Untriaged
-
-
0
-
Unknown
Description
Steps:
- Create 5 node fts cluster
- Load 60M vector data with 1536 dims
- Assign 16GB ram to each fts node
- Create 1 index with 90 partition (18 part/node)
Run 10QPS for 1 Min, following was observed
- when indexing reached 10M. 100% queries failed
- when indexing reached 60M. 100% queries failed
- Manually ran some queries via UI, 4-5 passed
- Repeated step 1, saw more than 95% queries failing.
Queries are failing not just with 429 BUT ALSO:
172.23.105.122: fts |
2024-06-12T21:41:36.468-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59 |
2024-06-12T21:41:36.469-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59 |
2024-06-12T21:41:36.470-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59 |
2024-06-12T21:41:36.471-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59 |
2024-06-12T21:42:55.995-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, could not read request body, indexName: bucket1.scope_0.index1 -- rest.ShowErrorBody() at rest.go:59...[truncated] |
Memory usage only spiked to like 3GB other 13GB were still available.
Attachments
Issue Links
- relates to
-
MB-62375 Rebalance out operation fails with 503 Service Unavailable
- Closed