Dramatically increasing latency of SET operations during rebalance tests

Rebalance-in, 3 -> 4, 1 bucket x 50M x 2KB, 10K ops/sec.
More details here:
https://raw.github.com/pavel-paulau/perfrunner/master/tests/reb_in_kv_50M_dgm.test

Performance report:
http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=apollo_250-961_8d4_rebalance&report=BaseRebalanceReport

Summary of samples collected during rebalance:

reb.describe()
count 243.000000
mean 21.825949
std 20.854682
min 0.445127
25% 10.203481
50% 19.634962
75% 29.129982
max 181.994200

Logs:
http://ci.sc.couchbase.com/job/apollo-64/577/artifact/

Number of ep_num_not_my_vbuckets is very small (considering total number of SET operations performed).

Notice that in this particular case we control rate of ops no matter how slow responses are. Thus you won't observe any drop in throughput.

For some reason latency of GET operations is not that high:

reb.describe()
Out[32]:
count 243.000000
mean 1.960755
std 1.853250
min 0.339985
25% 0.633001
50% 1.745939
75% 2.168894
max 12.428045