Details
-
Improvement
-
Resolution: Fixed
-
Major
-
5.0.0
Description
(1) In BackfillManager we run the scheduled backfills in a round robin manner.
If there is a pause in backfill scan due to full 'backfillBuffer' and/or full 'scanBuffer', this is probably fine with disk buckets because the items not scanned are still on disk. I am saying "probably" because this could still be an impediment (not sure if substantial) to the backfill perf as once scan is done we wait for other backfills to complete before transitioning to in-memory streaming.
In Ephemeral buckets, this is not desired because we will end up having more 'read snapshots' in case of high memory usage. That is, if 'backfillBuffer' and/or 'scanBuffer' is full, we pause one backfill (read snapshot still held) and run other backfills as well (more read snapshots).
(2) After problem (1) is solved we will have to revisit whether we need separate 'backfillBuffer' and 'scanBuffer' for ephemeral buckets. I think it would be best if solved by having 'BackfillManager' subclasses for ephemeral and couchbase buckets. For that, we might have to move the 'BackfillManager' is KVBucket subclasses and that might need some amount of refactoring.
Attachments
Issue Links
- relates to
-
MB-24713 Duration of rebalance with ephemeral buckets doubled in recent Spock builds
- Closed