Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0, 7.1.1
-
Untriaged
-
1
-
Yes
Description
What's the issue?
The worker pool used by the 'MPDownloader' is currently limited to a size of one, greatly reducing download performance.
pool := hofp.NewPool(hofp.Options{Size: 1, LogPrefix: "(objutil)"})
|
Running a download of a single 35GiB file on an EC2 instance with a gp3 EBS volume; using aws-cli, tools-common and tools-common with this fixed gives the following times/rates:
aws-cli: 4mins31.6s = 132.0MB/s
|
tools-common: 13mins26s = 44.5MB/s
|
tools-common (fixed): 4mins38.2s = 128.8MB/s
|
What's the fix?
We shouldn't be setting the 'Size' attribute.
Attachments
For Gerrit Dashboard: MB-52544 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
176372,2 | MB-52544 Don't set the download worker pool size to one | neo | tools-common | Status: MERGED | +2 | +1 |
176545,2 | MB-52544 Update tools-common dependency | neo | backup | Status: MERGED | +2 | +1 |
176546,2 | MB-52544 Update tools-common dependency | neo | cbbs | Status: MERGED | +2 | +1 |
176595,1 | Merge branch 'neo' into master | master | backup | Status: MERGED | +2 | +1 |
176597,1 | Merge branch 'neo' into master | master | cbbs | Status: MERGED | +2 | +1 |
176789,2 | Merge branch 'neo' into master | master | tools-common | Status: MERGED | +2 | +1 |