Set with Caching Hints

There are cases where an application has specific requirements with regards to the "importance" of having a specific item resident in a specific storage tier (memory, ssd, disk).

In some cases data is known to be extremely crucial to be available at low latency, so it should ideally never be evicted from RAM or evicted from SSD to spinning disk.

In other cases data might be known to only be of high importance if the same key is already in RAM at the time of the set (i.e. it only update the working set, but don't add new items to the cached in RAM set).

One example is a bulk update from a Hadoop cluster analytics job. The update would update large amounts of data, yet only a fraction of the data is in the current working set of the cluster at the point of importing the updated data. If the imported data will eject much of the working set present prior to the import, reads of old data set will now have to be serviced from disk, resulting in a lot of RAM misses after a new bulk import.

Proposed is a Set operation which carries an additional hint

Modes to consider are:
a) Always eject, don't try and keep in memory
b) Eject straight away if the key is not currently present in RAM
c) Never eject further than to SSD (if availabe)
d) Never eject from RAM

TBD is whether pinning operations (i.e. constraints on eviction) should be treated as hints (i.e. they always succeed, even if in the extreme another pinned item will need to be evicted in the case of RAM pressure) or whether they might fail, if no storage at the desired level is available anymore