As seen in rebalance test described in , a cluster with KV plus other services (FTS and GSI in the above instance) performing a (KV) rebalance can hang if KV requires backfills and all of the available backfill slots (default 4096) are consumed by other services.

For example, it was observed that 4231 streams were attempting to backfill:

Which were made up of:

However given there's only 4096 possible at once, a number of stream were pending (waiting for a slot to become available before they can start):

While we can also look at reducing the number of concurrent streams other services create, Ideally we want a solution such that KV is "defensive" - irrespective of what other services request, it can always make rebalance progress.