Details
-
Task
-
Resolution: Won't Do
-
Major
-
None
-
5.0.0
Description
Currently, there are 3 ways that expired items are cleaned up in the system:
- When a get request hit an expired item. During this time, this item will be deleted from the system
- Expiry pager. This is a background daemon process that runs periodically, or if memory usage reaches the high watermark. This process will look for keys that are expired in memory and clean up this key. This process is sufficient for value eviction policy because all keys/metadata are stored in RAM. On the other hand, in full eviction case, it may only look for a subset of expired keys in memory.
- During compaction. During compaction phrase, the compactor will copy all the documents in the old file and put them into the new file, during this process, it will also look through all expired items and delete them during the compaction progress.
In case of FDB integration project, FDB provides two compaction mode:
- Full compaction
- Circular block reuse. In this case, there’s a possibility that no full compaction will be ran for a very long time. As we discussed before, expiry pager is only sufficient for value eviction, in the case of full eviction(DGM) case, expiry pager will miss items whose keys are not kept in memory. Therefore, it’s possible that a large number of expired items may reside in disk for a very long time. Therefore, I propose that we extend expiry pager to look for expired keys in disk as part of this project.