Details
-
Improvement
-
Resolution: Done
-
Major
-
6.6.0
Description
The 'GetArchiveMeta' and 'PutArchiveMeta' are currently synchronous functions, we should make them get/put archive metadata concurrently because in theory these are long running, low CPU, IO blocking operations.
We should also make use of the AWS Uploaders/Downloaders to upload larger than normal files e.g. >=100MB concurrently. This should make the setup/tear-down significantly faster.
Both of these may be slightly more difficult to test locally (using a mock S3) since, from my experience they don't like having lot's of multipart uploads going on concurrently.