Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40531

HiFi MRU Item Pager evicts active items when 2-bit LRU does not

    XMLWordPrintable

Details

    Description

      Summary

      As seen during a CBSE investigation, when running on 6.5.1 the Item Pager ends up evicting items from active vBuckets, even when there is ample replica vBucket items which should be evicted first.

      The result of this is a significant performance regression compared to pre HiFi-MRU Item Pager (such as 5.1.3).

      Details

      Under the following workload / config:

      • Single bucket, 1 replica, value-only eviction bucket.
      • A workload which performs a mix of GET, APPEND and SET operations (so dataset size does gradually increase due to appends). Access pattern is uniform (all keys accessed with equal probability).
      • Bucket quota sized such that all active and replica items can initially be resident, but as dataset increases in size replica items can no longer be held 100% resident.
      • Workload runs at a fixed throughput, measuring latency of operations.

      When the workload starts and both active and replica are 100% resident, 6.5.1 exhibits better (lower) latency than 6.5.1. However, when the high watermark is reached (and some items must be evicted to bring memory back to low watermark), the latency observed on 6.5.1 is significantly worse:

      (Item Pager starts at ~10:57, when 6.5.1 latency starts to increase).

      This is not the case with 5.1.3 - while it's not explicitly shown on the graph, the ItemPager does start on 5.1.3 around the same time.

      Analysis

      Examining the residency ratio of active and replica vBuckets, we can see that in the 6.5.1 case the RR of active vBuckets frequently drops below 100%:

      This is not the case with 5.1.3, which maintains a 100% active RR even as the replica RR drops:

      Attachments

        1. 513_vBucket_RR.png
          513_vBucket_RR.png
          145 kB
        2. 651_vBucket_RR.png
          651_vBucket_RR.png
          105 kB
        3. cache_miss_after_fix.png
          cache_miss_after_fix.png
          460 kB
        4. cache_miss_before_fix.png
          cache_miss_before_fix.png
          410 kB
        5. hercules_4146_651_RR.png
          hercules_4146_651_RR.png
          127 kB
        6. hercules_4147_513_RR.png
          hercules_4147_513_RR.png
          123 kB
        7. hercules_4160_700_RR.png
          hercules_4160_700_RR.png
          116 kB
        8. image.png
          image.png
          105 kB
        9. image-2020-08-25-10-20-59-930.png
          image-2020-08-25-10-20-59-930.png
          172 kB
        10. image-2020-08-25-10-22-43-617.png
          image-2020-08-25-10-22-43-617.png
          186 kB
        11. image-2020-08-25-10-23-52-468.png
          image-2020-08-25-10-23-52-468.png
          214 kB
        12. image-2020-08-25-10-25-59-638.png
          image-2020-08-25-10-25-59-638.png
          235 kB
        13. image-2020-08-25-10-36-13-844.png
          image-2020-08-25-10-36-13-844.png
          239 kB
        14. image-2020-08-25-10-51-57-284.png
          image-2020-08-25-10-51-57-284.png
          482 kB
        15. image-2020-08-28-10-24-22-353.png
          image-2020-08-28-10-24-22-353.png
          190 kB
        16. image-2020-08-28-10-25-18-810.png
          image-2020-08-28-10-25-18-810.png
          195 kB
        17. image-2020-08-28-10-51-22-360.png
          image-2020-08-28-10-51-22-360.png
          217 kB
        18. image-2020-08-28-10-52-06-375.png
          image-2020-08-28-10-52-06-375.png
          235 kB
        19. image-2020-08-28-10-53-22-062.png
          image-2020-08-28-10-53-22-062.png
          265 kB
        20. image-2020-08-28-10-53-48-106.png
          image-2020-08-28-10-53-48-106.png
          312 kB
        21. image-2020-08-28-12-27-46-737.png
          image-2020-08-28-12-27-46-737.png
          360 kB
        22. image-2020-09-01-10-23-15-959.png
          image-2020-09-01-10-23-15-959.png
          206 kB
        23. Screenshot 2020-07-17 at 12.13.53.png
          Screenshot 2020-07-17 at 12.13.53.png
          185 kB

        Issue Links

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty