Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39137

cbbackupmgr reduce the Rift scan buffer size from 256MB to ~5MB

    XMLWordPrintable

Details

    Description

      The scan buffer size was chosen by me when I first wrote Rift; it was an arbitrary choice which was made in the moment. After looking at some of the performance testing on Leto, it's clear that we are buffer significantly more data in memory than is needed.

      I've done some local performance testing with various document sizes most importantly about 5GB worth of 1KB documents into an ephemeral bucket and haven't seen a significant drop in performance.

      On a restore/merge the bottleneck shouldn't be the full scan from disk it should be the restore worker pool/the Rift writer on the other end. We don't really need to buffer as much data as 256MB per-thread it's just overkill.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            james.lee James Lee
            james.lee James Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty