10B rebalance tests with Couchstore no-progress on build 7.0.0-3938

Note this was filed as MB-43288 with two sets of logs for what could be 2 issues, this is the second issue.

I ran 10B rebalance test with Couchstore for Magma baseline. My runs failed twice with different issues. Both runs were using build 7.0.0-3938.

My second run failed because the rebalance didn't make any progress for 6 hours. The run was terminated after it hit the 6-hour timeout threshold.

Job: http://perf.jenkins.couchbase.com/job/rhea-5node1/565/

Log: 

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-565/172.23.97.21.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-565/172.23.97.22.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-565/172.23.97.23.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-565/172.23.97.24.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-rhea-5node1-565/172.23.97.25.zip

2020-12-11T20:58:24 [INFO] Rebalance progress: 55.46922024623802 %

...

2020-12-12T02:58:24 [INFO] Rebalance progress: 55.46922024623802 %

2020-12-12T02:58:26 [ERROR] Rebalance hung