Critical Description
What's the issue?
We're starting to get to the point now, where "listing" the contents of remote archives is becoming a significant performance issue, listing ~2 million items in GCP takes ~20 minutes.
This is an issue for lots of the workflows in 'cbbackupmgr', for example:
1) Deleting a directory
2) Populating the staging directory
3) Listing the parts for in-progress multipart uploads
We need to investigate what the top-end we want to support is, document it and potentially add some guards into to stop users from going over the recommended amount.
What's the fix?
1) We need to determine how many backups can be created, whilst allowing reasonable listing performance
2) We should add some guards into 'cbbackupmgr' to detect/error out if too many backups are created
3) We should improve the backup service to allow it to handle this case by creating a new repository (which limits the scope for objects - except for archive level sub-commands e.g. collect-logs)
Attachments
Issue Links
- relates to
-
MB-60677 [CBM] Investigate whether further gains can be made in staging directory performance
-
- Open
-
-
-
- Resolved
-
-
DOC-12119 Add recommendation to constrain the size of backup repositories
-
- Open
-
-
-
- Open
-
-
MB-60678 [CBM] Increase staging directory lockfile polling
-
- Open
-
-
-
- Open
-
- links to