Details
-
Task
-
Resolution: Won't Do
-
Critical
-
4.0.0
-
Security Level: Public
-
None
Description
On a 4KV + 1 indexer cluster, if I add another indexer (physical machine) and shard/partition the index over the two indexer nodes (each indexer node gets one index with a predicate that cuts the index's key space in half) vs. putting these two indexes on the same indexer box (sharding within one indexer node), we see the following results compared to baseline. Note, 53% here means it takes 53% of the original/baseline time.
initial build Incremental build
2 indexers 53.27% 61.51%
1 indexer 2 shards 62.31% 81.20%
link to full data: https://docs.google.com/document/d/1v620uzoWIGRFIZY4bdSzj5IYnpTRsstPtxw30GApJ5Q/edit?usp=sharing
We should investigate why 81.20% isn't closer to 50%, ie., why doesn't it scale as linearly as the others?