Major Description
While the system test is in progress, the indexing stopped while the cluster was rebalancing and the index ops was zero. The CPU usage and RAM usage also is not high still the indexing was stopped/paused.
Steps:
- Created an on-prem cluster with 12 nodes. Out of which 5 are fts nodes.
- Each node has atleast 12gb of memory.
- Created one scope and one collection under that scope
- Loaded 5 million documents for normal vectors and 10k documents with xattrs using the sift dataset
- Created two indexes with 90 partitions acoss 5 fts nodes. One index indexes the normal vector data and the other index indexes the xattrs vectors.
- Run knn queries
- Mutated the documents and then again ran the knn query
- Rebalanced in an FTS node
- Perform mutations and then again run the knn queries
- Rebalanced out an FTS node
Logs:
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.105.122.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.106.176.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.106.30.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.96.198.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.96.230.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.96.245.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.100.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.108.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.109.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.66.zip
- https://cb-engineering.s3.amazonaws.com/system-test-koushal-cluster/collectinfo-2024-05-17T041448-ns_1%40172.23.97.67.zip
Attached are two images. One shows the search page where the indexing rate is zero and the second image shows the resource utilization for the cluster nodes.
