Description
The current approach to ConfigChanged can contribute to thread pool depletion. ConfigHandler.Process is running on the thread pool firing synchronous ConfigUpdated events. However, the event handlers are starting async tasks and then awaiting them, blocking the thread from the thread pool where ConfigHandler.Process is running.
Additionally, since ConfigHandler.Process is a synchronous method with no awaits, I believe it will permanently use a thread from the thread pool for the lifetime of the cluster.
I see two possible solutions. One, make ConfigHandler.Process async all the way down the stack to the ConfigUpdated event handlers (including the queue wait). The second would be to flag the ConfigHandler.Process task with TaskCreationOptions.IsLongRunning, which causes the TPL to allocate a dedicated thread.