Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52544

[TC] [Downloader] MPDownloader only uses a single worker to download objects

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.1.2
    • 7.1.0, 7.1.1
    • tools
    • Untriaged
    • 1
    • Yes

    Description

      What's the issue?
      The worker pool used by the 'MPDownloader' is currently limited to a size of one, greatly reducing download performance.

      pool := hofp.NewPool(hofp.Options{Size: 1, LogPrefix: "(objutil)"})

      Running a download of a single 35GiB file on an EC2 instance with a gp3 EBS volume; using aws-cli, tools-common and tools-common with this fixed gives the following times/rates:

      aws-cli: 4mins31.6s = 132.0MB/s
      		 
      tools-common: 13mins26s = 44.5MB/s
      		 
      tools-common (fixed): 4mins38.2s = 128.8MB/s

      What's the fix?
      We shouldn't be setting the 'Size' attribute.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            gilad.kalchheim Gilad Kalchheim
            james.lee James Lee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty