Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.6.0
-
Untriaged
-
1
-
No
Description
The scan buffer size was chosen by me when I first wrote Rift; it was an arbitrary choice which was made in the moment. After looking at some of the performance testing on Leto, it's clear that we are buffer significantly more data in memory than is needed.
I've done some local performance testing with various document sizes most importantly about 5GB worth of 1KB documents into an ephemeral bucket and haven't seen a significant drop in performance.
On a restore/merge the bottleneck shouldn't be the full scan from disk it should be the restore worker pool/the Rift writer on the other end. We don't really need to buffer as much data as 256MB per-thread it's just overkill.