Minor Description
When looking into a forum issue around backups to S3 we noticed that the performance of populating the staging directory appeared to be an issue. There have been a couple of improvements recently (see linked MBs) but there could be gains still to be made.
To give an idea of the severity of the problem: some testing on developer machines suggests that populating the staging directory (which includes listing all the files, downloading the ones we need and then unzipping metadata files) would take ~20 minutes for a repository with 365 backups in (i.e. a year's worth of daily backups). It's worth noting that this is an extreme case - we wouldn't recommend a user do this but we have seen similar workloads in the wild. For example the forum post was a week's worth of hourly incrementals (168 backups), which would take ~9 minutes.
Initial investigation suggests that on a 26 backup repository it took around 3.31s to populate a single backup's files - ~2.31s being listing/downloading and ~1s for unzipping the range, failover log, snapshot and stats files. On mac the latter is far worse - probably due to worse IO performance.
Some initials ideas:
- We currently list files sequentially to discover what repos exist. We could do this concurrently
- We have a worker pool for unzipping, but only at the backup level. We could unzip the files for multiple backups concurrently. Unzipping four files at once seems to use very little CPU, so that is not too much of a worry
