Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
None
-
Untriaged
-
1
-
No
-
KV 2021-Dec, KV 2022-Jan
Description
Items may be expired by compaction, or for stored values which are in the hashtable, by a PagingVisitor (created by ExpiredItemPager and ItemPager).
(They may also be expired on access, but this is not discussed here).
The callback used during compaction to expire items checks memory usage is below a configureable quota.
class ExpiredItemsCallback : public Callback<Item&, time_t&> {
|
...
|
void callback(Item& it, time_t& startTime) override {
|
if (epstore.compactionCanExpireItems()) {
|
epstore.deleteExpiredItem(it, startTime, ExpireBy::Compactor);
|
}
|
}
|
This prevents compaction driving memory usage arbitrarily high; the default threshold is 85%, the same as the default high watermark.
Expiry via PagingVisitor does not make any checks before expiring items.
In Neo, the CheckpointManager quota has been introduced. This is used to limit the maximum memory CheckpointManagers for all vbuckets can consume.
As this is not checked in either expiry path, checkpoint memory usage may be driven beyond the CM quota (if it cannot be reduced by checkpoint removai/expelling/cursor dropping fast enough).
Expiry through either compaction or visitor should consider the state of the CM, as a front end op would.
Attachments
Issue Links
- relates to
-
MB-49042 [Magma] - Items not in memory don't expire in storage after full compaction(compaction-callback) is triggered multiple times.
- Closed