HiFi MRU Item Pager evicts active items when 2-bit LRU does not

Summary

As seen during a CBSE investigation, when running on 6.5.1 the Item Pager ends up evicting items from active vBuckets, even when there is ample replica vBucket items which should be evicted first.

The result of this is a significant performance regression compared to pre HiFi-MRU Item Pager (such as 5.1.3).

Details

Under the following workload / config:

• Single bucket, 1 replica, value-only eviction bucket.
• A workload which performs a mix of GET, APPEND and SET operations (so dataset size does gradually increase due to appends). Access pattern is uniform (all keys accessed with equal probability).
• Bucket quota sized such that all active and replica items can initially be resident, but as dataset increases in size replica items can no longer be held 100% resident.
• Workload runs at a fixed throughput, measuring latency of operations.

When the workload starts and both active and replica are 100% resident, 6.5.1 exhibits better (lower) latency than 6.5.1. However, when the high watermark is reached (and some items must be evicted to bring memory back to low watermark), the latency observed on 6.5.1 is significantly worse:

(Item Pager starts at ~10:57, when 6.5.1 latency starts to increase).

This is not the case with 5.1.3 - while it's not explicitly shown on the graph, the ItemPager does start on 5.1.3 around the same time.

Analysis

Examining the residency ratio of active and replica vBuckets, we can see that in the 6.5.1 case the RR of active vBuckets frequently drops below 100%:

This is not the case with 5.1.3, which maintains a 100% active RR even as the replica RR drops: