views latency increase after perfrunner change

Recently in perfrunner I made a change to how our KV ops are throttled. This change made the KV op rate much more stable around the desire operations/sec. For example before the change, if we wanted 100 KV update ops/sec, the actual rate would fluctuate around 100 by jumping up to 175, then down to 50. Now the rate is much closer to 100 ops/sec through out the whole test.

This has caused some strange behaviour in a subset of the views performance tests. Here is the showfast page with all the affected tests: http://showfast.sc.couchbase.com/#/timeline/Linux/views/by_type/all

Latency for these tests have jumped up by 4-5x

Looking at a specific test:

80th percentile query latency (ms), 1 node, 20M docs, 3 views, 100 updates/sec, 100 queries/sec, stale=false, basic queries

7.0.0-2588: 

http://perf.jenkins.couchbase.com/job/leto/13999/ - 58.8

http://perf.jenkins.couchbase.com/job/leto/14131/ - 268

Comparison graphs (green is before, orange is after change):

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=leto_700-2588_access_d1b5&snapshot=leto_700-2588_access_ec86

In the graphs, you will notice "couch_views_actual_disk_size" and "couch_views_fragmentation" grows at a constant rate despite compactions being set to 30%