Loading...

Details

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: 7.6.4
Affects Version/s: 7.6.2
Component/s: fts
Labels:
- vector-search
Environment:
Enterprise Edition 7.6.2 build 3710

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.106.171.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.67.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.105.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.106.171.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.106.30.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.96.198.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.96.230.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.96.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.100.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.108.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.109.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.66.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1718265625/collectinfo-2024-06-13T080218-ns_1%40172.23.97.67.zip
Story Points:
0
Is this a Regression?:
Unknown

Description

Steps:

Create 5 node fts cluster
Load 60M vector data with 1536 dims
Assign 16GB ram to each fts node
Create 1 index with 90 partition (18 part/node)

Run 10QPS for 1 Min, following was observed

when indexing reached 10M. 100% queries failed
when indexing reached 60M. 100% queries failed
Manually ran some queries via UI, 4-5 passed
Repeated step 1, saw more than 95% queries failing.

Queries are failing not just with 429 BUT ALSO:

172.23.105.122: fts

2024-06-12T21:41:36.468-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59

2024-06-12T21:41:36.469-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59

2024-06-12T21:41:36.470-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59

2024-06-12T21:41:36.471-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, indexName: bucket1.scope_0.index1, err: pindex_consistency: ConsistencyWaitGroup cancelled -- rest.ShowErrorBody() at rest.go:59

2024-06-12T21:42:55.995-07:00 [ERRO] rest: error code: 400, msg: rest_index: Query, could not read request body, indexName: bucket1.scope_0.index1 -- rest.ShowErrorBody() at rest.go:59...[truncated]