Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40531

HiFi MRU Item Pager evicts active items when 2-bit LRU does not

    XMLWordPrintable

    Details

      Description

      Summary

      As seen during a CBSE investigation, when running on 6.5.1 the Item Pager ends up evicting items from active vBuckets, even when there is ample replica vBucket items which should be evicted first.

      The result of this is a significant performance regression compared to pre HiFi-MRU Item Pager (such as 5.1.3).

      Details

      Under the following workload / config:

      • Single bucket, 1 replica, value-only eviction bucket.
      • A workload which performs a mix of GET, APPEND and SET operations (so dataset size does gradually increase due to appends). Access pattern is uniform (all keys accessed with equal probability).
      • Bucket quota sized such that all active and replica items can initially be resident, but as dataset increases in size replica items can no longer be held 100% resident.
      • Workload runs at a fixed throughput, measuring latency of operations.

      When the workload starts and both active and replica are 100% resident, 6.5.1 exhibits better (lower) latency than 6.5.1. However, when the high watermark is reached (and some items must be evicted to bring memory back to low watermark), the latency observed on 6.5.1 is significantly worse:

      (Item Pager starts at ~10:57, when 6.5.1 latency starts to increase).

      This is not the case with 5.1.3 - while it's not explicitly shown on the graph, the ItemPager does start on 5.1.3 around the same time.

      Analysis

      Examining the residency ratio of active and replica vBuckets, we can see that in the 6.5.1 case the RR of active vBuckets frequently drops below 100%:

      This is not the case with 5.1.3, which maintains a 100% active RR even as the replica RR drops:

        Attachments

        1. 513_vBucket_RR.png
          513_vBucket_RR.png
          145 kB
        2. 651_vBucket_RR.png
          651_vBucket_RR.png
          105 kB
        3. cache_miss_after_fix.png
          cache_miss_after_fix.png
          460 kB
        4. cache_miss_before_fix.png
          cache_miss_before_fix.png
          410 kB
        5. hercules_4146_651_RR.png
          hercules_4146_651_RR.png
          127 kB
        6. hercules_4147_513_RR.png
          hercules_4147_513_RR.png
          123 kB
        7. hercules_4160_700_RR.png
          hercules_4160_700_RR.png
          116 kB
        8. image.png
          image.png
          105 kB
        9. image-2020-08-25-10-20-59-930.png
          image-2020-08-25-10-20-59-930.png
          172 kB
        10. image-2020-08-25-10-22-43-617.png
          image-2020-08-25-10-22-43-617.png
          186 kB
        11. image-2020-08-25-10-23-52-468.png
          image-2020-08-25-10-23-52-468.png
          214 kB
        12. image-2020-08-25-10-25-59-638.png
          image-2020-08-25-10-25-59-638.png
          235 kB
        13. image-2020-08-25-10-36-13-844.png
          image-2020-08-25-10-36-13-844.png
          239 kB
        14. image-2020-08-25-10-51-57-284.png
          image-2020-08-25-10-51-57-284.png
          482 kB
        15. image-2020-08-28-10-24-22-353.png
          image-2020-08-28-10-24-22-353.png
          190 kB
        16. image-2020-08-28-10-25-18-810.png
          image-2020-08-28-10-25-18-810.png
          195 kB
        17. image-2020-08-28-10-51-22-360.png
          image-2020-08-28-10-51-22-360.png
          217 kB
        18. image-2020-08-28-10-52-06-375.png
          image-2020-08-28-10-52-06-375.png
          235 kB
        19. image-2020-08-28-10-53-22-062.png
          image-2020-08-28-10-53-22-062.png
          265 kB
        20. image-2020-08-28-10-53-48-106.png
          image-2020-08-28-10-53-48-106.png
          312 kB
        21. image-2020-08-28-12-27-46-737.png
          image-2020-08-28-12-27-46-737.png
          360 kB
        22. image-2020-09-01-10-23-15-959.png
          image-2020-09-01-10-23-15-959.png
          206 kB
        23. Screenshot 2020-07-17 at 12.13.53.png
          Screenshot 2020-07-17 at 12.13.53.png
          185 kB

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-4258 contains kv_engine commit 11957d0 with commit message:
            MB-40531:[BP] Allow VBucketVisitors to specify vbucket order

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4258 contains kv_engine commit 11957d0 with commit message: MB-40531 : [BP] Allow VBucketVisitors to specify vbucket order
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-4258 contains kv_engine commit 6c72cfc with commit message:
            MB-40531: [BP] Prefer paging from replicas if possible

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4258 contains kv_engine commit 6c72cfc with commit message: MB-40531 : [BP] Prefer paging from replicas if possible
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-4258 contains kv_engine commit c73d901 with commit message:
            MB-40531: [BP] Make replica eviction ignore age threshold

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4258 contains kv_engine commit c73d901 with commit message: MB-40531 : [BP] Make replica eviction ignore age threshold
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9447 contains kv_engine commit 1dfec49 with commit message:
            MB-40531: [BP] Make replica eviction ignore age threshold

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9447 contains kv_engine commit 1dfec49 with commit message: MB-40531 : [BP] Make replica eviction ignore age threshold
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-4322 contains kv_engine commit 1dfec49 with commit message:
            MB-40531: [BP] Make replica eviction ignore age threshold

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4322 contains kv_engine commit 1dfec49 with commit message: MB-40531 : [BP] Make replica eviction ignore age threshold

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There is 1 open Gerrit change

                    PagerDuty