Vector search sizing exercise

Test 1

1. Load 30 million vector dataset documents.
2. Create a single FTS index with 1 partition per index per node.
3. Wait for index building to complete.

Observation

We start seeing OOM issues after indexing around 17 million documents.
Logs - http://supportal.couchbase.com/snapshot/7340949163006b165a226742c5d21564::0

Test 2

1. Load 20 million vector dataset documents.
2. Create a single FTS index with 1 partition per index per node.
3. Wait for index building to complete.

Observation

Out of memory exceptions seen after indexing 19 million documents or so.
Logs - http://supportal.couchbase.com/snapshot/2bd413c57b16b476939cbd14053d61e8::0

Test 3

1. Load 2 million vector dataset documents.
2. Create a single FTS index with 1 partition per index per node, total 5 partitions.
3. Wait for index building to complete.
4. Insert incremental workload of 40 million items at a constant rate of 2k ops per sec.

Observation

FTS service is continuously crashing in a loop which has caused index building process to hang after indexing 33 million items or so.
Logs - http://supportal.couchbase.com/snapshot/2bd413c57b16b476939cbd14053d61e8::2

Test 4

1. Load 30 million vector dataset documents.
2. Create a single FTS index with 8 partitions per index per node, total 40 partitions.
3. Wait for index building to complete.
4. Insert incremental workload of 40 million items at a constant rate of 2k ops per sec.

Observation

FTS went into a crash loop after 50 million documents. FTS nodes getting AF'ed over continuously due to OOM issues and control plane adds the node back into the cluster and this cycle is getting repeated continuously.
Logs - http://supportal.couchbase.com/snapshot/7340949163006b165a226742c5d21564::3

19/20th Feb 384 dimensions test runs

CASE A

Logs -

CASE B

Logs -

CASE C

Logs -