Details
-
Bug
-
Resolution: Duplicate
-
Test Blocker
-
7.6.2
-
7.6.2-3688
-
Untriaged
-
0
-
Unknown
Description
Rebalance failed for the cluster with following error.
Rebalance exited with reason {service_rebalance_failed,fts,
|
{agent_died,<34981.3228.183>, |
{lost_connection,
|
{'ns_1@172.23.106.30',shutdown}}}}. |
Rebalance Operation Id = 7ae982048365c414e51ffd073833cb10
|
Please refer to the logs for more details.
Steps:
- Created an on-prem cluster with 12 nodes. Out of which 5 are fts nodes.
- Loaded 100million documents. Each node has atleast 12gb of memory.
- Created one scope and four collections under that scope
- coll_0: Normal Vector Data (5m documents)
- coll_1: Vector data with xattr (1m documents)
- coll_2: Vector data with base_64 (1m documents)
- coll_3: Vector data (1m documents)
- Creating four indexes with 12 partitions across 5 fts nodes.
- index_0: Indexing Normal Vector Data (5m documents)
- index_1: Indexing Vector data with xattr (1m documents)
- index_2: Indexing Vector data with base_64 (1m documents)
- index_3: Indexing Vector data (1m documents)
- Run 50 parallel knn queries on all the indexes created
- Mutated the documents and then again run the 50 parallel knn query on all the indexes
- Add an FTS node
- Perform mutations and then again run the 50 parallel knn queries on all the indexes
- Remove an FTS node
The above steps are performed in a loop.
Logs:
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.106.176.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.106.30.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.96.198.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.96.230.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.96.245.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.97.100.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.97.66.zip
- https://cb-engineering.s3.amazonaws.com/ashok-koushal-system-test-oom/collectinfo-2024-06-03T025946-ns_1%40172.23.97.67.zip
Could get the logs only of the above nodes.
Attachments
Issue Links
- relates to
-
MB-62019 Service 'fts' exited with status 2 while running queries
- Closed