Investigate LRU efficiency when replicated in read-heavy workloads

Scenario

In DGM scenarios with a high read rate, the per-document LRU information between active and replica vBuckets can become mismatched. This is because LRU information is only replicated in DCP_MUTATION messages - if there are no mutations then no LRU state is replicated. As such, the set of documents which are resident in the active and replica(s) of a given vBucket can vary significantly.

In the event of a failover; the replica vBuckets which have been promoted to active can exhibit a much higher cache miss ratio.

Suggestions

• Change how LRU LRU information is replicated - not just when documents change (via DCP_MUTATION):
  • For hottest documents; periodically replicate their current LRU via a new DCP message
• Prioritise some replicas over others - take into consideration how many replicas there are and evict the 2nd and 3rd replicas over the 1st.