Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38797

cbbackupmgr limit the number of threads for cloud backup

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Done
    • Major
    • 6.6.0
    • 6.6.0
    • tools

    Description

      The threading model has changed slightly with the introduction of cloud based backup; the number of threads the user supplies to cbbackupmgr directly correlates to how many vBuckets are backed up at once.

      When doing local backups it's generally accepted that using about the number of vCPU threads is a good idea; if you use too many you will end up thrashing the filesystem cache and it will be a detriment to the cbbackupmgr (cbbackupmgr is disk IO bound).

      Native cloud integration is in a similar boat, except we are bound by the network bandwidth; local testing has shown that using more threads is significantly better for native cloud backups (most of the time the threads are asleep on IO wait - and we are using Go so goroutines are cheap).

      We know up front:
      1) How many threads the user wishes to use
      2) How much memory the system has
      3) How much data will be buffered in memory by the object store data writer

      With this data we can limit the number of threads cbbackupmgr uses (either by rejecting invalid choices or by using max(user choice, calculated max). This will help to avoid cases where (with poor network bandwidth) it would be possible to exhaust system memory buffering data from the cluster.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            owend Daniel Owen
            james.lee James Lee
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty