Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24681

Ephemeral Backfill Memory Management: Finish one backfill completely before proceeding to other backfills

    XMLWordPrintable

Details

    Description

      (1) In BackfillManager we run the scheduled backfills in a round robin manner.

      If there is a pause in backfill scan due to full 'backfillBuffer' and/or full 'scanBuffer', this is probably fine with disk buckets because the items not scanned are still on disk. I am saying "probably" because this could still be an impediment (not sure if substantial) to the backfill perf as once scan is done we wait for other backfills to complete before transitioning to in-memory streaming.

      In Ephemeral buckets, this is not desired because we will end up having more 'read snapshots' in case of high memory usage. That is, if 'backfillBuffer' and/or 'scanBuffer' is full, we pause one backfill (read snapshot still held) and run other backfills as well (more read snapshots). 

      (2) After problem (1) is solved we will have to revisit whether we need separate 'backfillBuffer' and 'scanBuffer' for ephemeral buckets. I think it would be best if solved by having 'BackfillManager' subclasses for ephemeral and couchbase buckets. For that, we might have to move the 'BackfillManager' is KVBucket subclasses and that might need some amount of refactoring.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              manu Manu Dhundi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty