This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
- System should have more than 1 functions deployed on the same src-bkt
- KV rebalance should be triggered
- As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt
- If system hits this race condition, rebalance will be stuck and will fail after 10 mins.
customer deployment ran into this issue twice.