Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.6.0
-
0
-
Yes
-
KV 2023-4
Description
Currently, we have no checks if the bucket quota will be exceeded when a new ActiveStream is created. This can cause memory usage to exceed the bucket quota in certain situations.
For example, I created a single node cluster with 1GB of memory usage and 999 collections. Then I ran dcpdrain with a stream config that maps each collection to a unique ActiveStream (mimicing CBAS). This creates 999 * 1024 = 1,022,976 ActiveStreams in total and takes up a total of ~1GB of bucket quota just to manage all those connections. Because no check is included to validate if the bucket quota will be exceeded when a new ActiveStream is created, the total memory used increases above the bucket quota. As seen in my testing attached.
This issue is noticeable in small bucket quota environments intending to create a large number of ActiveStreams. Therefore, to prevent exceeding the Bucket quota we should add a sanity check to make sure the bucket quota will not be exceeded when creating a new ActiveStream.
Attachments
Issue Links
- causes
-
MB-59826 [Magma] - Rebalance out with 1 DGM bucket fails with "'Rebalance exited with reason {mover_crashed"
- Closed