Can't swap to old hash table eviction policy at runtime

Summary

With the introduction of the new hash table eviction policy (hifi_mfu) we provided the ability to switch back to the old policy (2bit_lru) in case an issue was found in the field.

Unfortunately although the switch can be made if the node is restarted - it currently cannot be done at runtime.

Details

The item pager relies on a phase.

The valid phases for the hifi_mfu eviction policy are REPLICA_ONLY and ACTIVE_AND_PENDING_ONLY.

The valid phases for the old 2bit_lru policy are PAGING_UNREFERENCED and PAGING_RANDOM,

With the 2bit_lru policy the phase is initialised to PAGING_UNREFERENCED. However for the hifi_mfu policy is it is initialised to REPLICA_ONLY or ACTIVE_AND_PENDING_ONLY depending on whether it is a persistent bucket or not.

Once the item pager is created for the hifi_mfu it can only swap between the REPLICA_ONLY or ACTIVE_AND_PENDING_ONLY phases, and therefore if the policy is changed dynamically to the 2bit_lru then it will not be in a valid phase.