Rebalance from 8 to 6 nodes is 1.5x slower when RZA is used

This is something that we already discussed but I was asked to make this public.

There are basically two tests:

1. I'm comparing 8->6 rebalance out, 2 replicas, 2.2.0 vs 2.5.0. As expected in this particular case number of movements is smaller (~1.5x) and rebalance is faster (33.9 minutes vs 19.8 minutes).

2.2.0 logs:
http://ci.sc.couchbase.com/job/apollo-64/631/artifact/

2.5.0 logs:
http://ci.sc.couchbase.com/job/apollo-64/632/artifact/

2. The same test but servers are distributed among multiple groups (332 -> 33). Number of movements does increase but rebalance disproportionately slower (19.8 minutes vs. 29.9 minutes). There was strange slowdown at the end of rebalance:

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=apollo_250-994_6dd_rebalance&snapshot=apollo_250-994_3d3_rebalance&report=BaseRebalanceReport

So maybe this is occasional issue.

Logs:
http://ci.sc.couchbase.com/job/apollo-64/633/artifact/