The amount of time a backfill may run for before yielding (allowing other backfills to run) is indirectly limited by:
- a per-scan byte limit
- outstanding bytes limit (bytes read by backfill but not yet sent by the stream)
However, for a collection-filtered stream, these limits are only applied to items which actually match the filter. A single collection may only be a small fraction of the data on disk - a seqno-ordered backfill may read a significant amount of data (taking a significant amount of time) before enough items for the desired collection have been read.
This may adversely affect other backfills for the same backfill manager, and other task running on AuxIO threads.
investigate altering the scan limit to account for bytes read even if they do not match the filter, or an additional limit solely tracking bytes or items read.
Uncovered during investigation for