Details
-
Improvement
-
Resolution: Done
-
Major
-
6.6.0
Description
The threading model has changed slightly with the introduction of cloud based backup; the number of threads the user supplies to cbbackupmgr directly correlates to how many vBuckets are backed up at once.
When doing local backups it's generally accepted that using about the number of vCPU threads is a good idea; if you use too many you will end up thrashing the filesystem cache and it will be a detriment to the cbbackupmgr (cbbackupmgr is disk IO bound).
Native cloud integration is in a similar boat, except we are bound by the network bandwidth; local testing has shown that using more threads is significantly better for native cloud backups (most of the time the threads are asleep on IO wait - and we are using Go so goroutines are cheap).
We know up front:
1) How many threads the user wishes to use
2) How much memory the system has
3) How much data will be buffered in memory by the object store data writer
With this data we can limit the number of threads cbbackupmgr uses (either by rejecting invalid choices or by using max(user choice, calculated max). This will help to avoid cases where (with poor network bandwidth) it would be possible to exhaust system memory buffering data from the cluster.