Details
-
Improvement
-
Resolution: Fixed
-
Major
-
7.2.0
-
0
-
Magma June-16 2023, Magma July-27 2023
Description
In the current 7.2 CDC functionality, the on-disk history retention "window" keeps every version written by KV upto the configured retention limits (i.e. size and/or time).
We could envisage a bucket configuration where there is only 1 collection which actually enables history and as such never deduplicates. However if there are other collections configured (with history=false), and those other collections are dominating the workload the history retention window can become dominated/consumed by those non-history collections.
E.g. If the user thinks they have 15 minutes of history (capped at 1GB), they may not actually have the last 15 minutes of their history collection.
We've discussed this issue before but never committed to solving it in 7.2 unless time allowed, but allowing the history retention window to be "partially" compacted (removal of non-historical collections) could be very useful to the overall feature.
Attachments
Issue Links
- relates to
-
MB-55931 [Magma-CDC] Magma is holding longer duration history than the configured hisotry_retention_seconds (since last sstable has not reached 5% of the total data, so it keeps holding data)
- Open