Description
While the system test is in progress, multiple rebalances were not successful. Also, if the rebalance is in progress, it takes lot of time to complete. The resources for the cluster are also not high when the rebalance is in progress. Refer to the image attached.
Steps:
- Created an on-prem cluster with 12 nodes. Out of which 5 are fts nodes.
- Each node has atleast 12gb of memory.
- Created one scope and one collection under that scope
- Loaded 5 million documents for normal vectors and 10k documents with xattrs using the sift dataset
- Created two indexes with 90 partitions acoss 5 fts nodes. One index indexes the normal vector data and the other index indexes the xattrs vectors.
- Run knn queries
- Mutated the documents and then again ran the knn query
- Rebalanced in an FTS node
- Perform mutations and then again run the knn queries
- Rebalanced out an FTS node
Logs:
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.105.122.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.106.176.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.106.30.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.96.198.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.96.230.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.96.245.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.100.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.108.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.109.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.66.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.67.zip