the rebalance time for the same build is dramatically different

In the rebalance-swap + durability majority tests, we observed that the run time varies a lot. For instance, the following two tests were running on the same build (7.0.0-1834), but they have very different run time. The difference is coming from how much time it took to finish rebalance. The rebalance time should not be dramatically different for same builds. We found the same issue happened on different builds.

Build: 7.0.0-1834

http://perf.jenkins.couchbase.com/job/titan-durability/67/

run time: 2 hr 27 min

2020-04-19T18:30:47 [INFO] Starting rebalance

2020-04-19T20:41:19 [INFO] Rebalance completed

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.100.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.101.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.102.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.103.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.104.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.105.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.106.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.107.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.108.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-67/172.23.96.109.zip

http://perf.jenkins.couchbase.com/job/titan-durability/65/

run time: 1 hr 33 min

2020-04-19T04:00:05 [INFO] Starting rebalance

2020-04-19T05:16:18 [INFO] Rebalance completed

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.100.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.101.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.102.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.103.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.104.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.105.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.106.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.107.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.108.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-durability-65/172.23.96.109.zip