Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
5.0.0
-
None
Description
Customers want to store billions of documents with tens or hundreds of items in an array index.
Our current storage scheme results in a very large index which directly impacts TCO / Hardware Requirements (Memory/Nodes etc.)
Creating this ticket as a placeholder for tracking the improvement in Spock.Next
As an example for 100 milliion documents, with 100 elements in an array each object of 10 bytes and a DocId size of 20 bytes. If you are using 128 GB of RAM and plan for 20% headroom for growth you are going to need 51 nodes for this single index without HA. This is just non tenable from a TCO standpoint - it excludes us from a lot of usecases
We need to fix this for both MOI and Plasma