The recent rework of OrphanReporter to prevent blocking of threads in the thread pool has created a slow memory leak. This is due to this corner-case bug in System.Threading.Channels: https://github.com/dotnet/runtime/issues/761
The queue within OrphanReporter is a very low volume queue, in ideal scenarios never receiving any items at all. In this case, using ChannelReader.WaitForReadyAsync with a timeout via CancellationToken will leak the memory associated with the CancellationTokenSource/callbacks/etc on each loop. This memory will be cleaned up whenever the next item arrives in the queue, but this may never happen and it will usually grow indefinitely.
I can confirm that this leak would also leak OperationCanceledExceptions. The core leak is around AsyncOperation<bool> instances, which then act to root several other objects so they can't be GC'd, including the OperationCanceledException, ExceptionDispatchInfo, CancellationTokenSource, CallbackNode, etc.
Jeffry Morris November 4, 2021 at 4:12 AM
Edited
In addition to this (or perhaps as a result of it) it seems like OperationCanceledExceptions are accumulated via ExceptionDispatchInfo and a reference back to Deque.
I saw this profiling tonight while investigating - OperationCanceledExceptions were the largest number objects allocated.
Fixed
Pinned fields
Click on the next to a field label to start pinning.
The recent rework of OrphanReporter to prevent blocking of threads in the thread pool has created a slow memory leak. This is due to this corner-case bug in System.Threading.Channels: https://github.com/dotnet/runtime/issues/761
The queue within OrphanReporter is a very low volume queue, in ideal scenarios never receiving any items at all. In this case, using ChannelReader.WaitForReadyAsync with a timeout via CancellationToken will leak the memory associated with the CancellationTokenSource/callbacks/etc on each loop. This memory will be cleaned up whenever the next item arrives in the queue, but this may never happen and it will usually grow indefinitely.
https://forums.couchbase.com/t/memory-leak-in-couchbase-net-sdk/32084