cbbackupmgr - compact speed degradation on ForestDB backups

Description

We're seeing a ~100% increase in compact time between 6.0.1/6.0.2 and the latest MadHatter builds. 
 
Test

EE compact time elapsed (seconds), 4 nodes, 1 bucket x 100M x 1KB, DGM, Idle

Results

6.5.0-3274: 1,294 secs

6.0.2-2409: 610 secs

 
Logs for 6.0.2-2409

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-10042/leto-srv-01.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-10042/leto-srv-02.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-10042/leto-srv-03.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-10042/leto-srv-04.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-10042/tools.zip

 
Logs for 6.5.0-3274

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-tmp1-52/leto-srv-01.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-tmp1-52/leto-srv-02.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-tmp1-52/leto-srv-03.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-tmp1-52/leto-srv-04.perf.couchbase.com.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-leto-tmp1-52/tools.zip

Comments

I would imagine this is to do with us having to open many more ForestDB files due to sharding changes. The logs indicate that throughout compact's runtime it is opening and closing many more ForestDB files than in prior versions. 

It should also be noted that this increase in compact time is offset by a large reduction in base backup size thanks to improvements in backup.