performance on non-IO related requests seems to drop during IO

When writing a quick test to check something out, I happened to write a test that would generate 'noise' with sets only from some threads and then do the actual work that I needed to test from some other threads.

The test had a number of threads doing small sets and a small number of threads doing sets with durability requirements. The sets with durability requirements will have the client do the set, then poll with the observe operation until the set has persisted to disk.

At no time would the test actually run out of memory. No TMPFAIL responses were received.

I happened to notice that as IO kicked in, the noisemaker threads were slowing down. This seems to correlate with when compaction kicked in at the server side. That shouldn't be the case though, as those threads are separate and frontend threads doing only sets, when memory is available, should not be slowed down this dramatically.

For example, the test was written such that the noisemaker threads should consistently generate about 15k ops/s IIRC. It was throttled at that rate by having a sleep time around an average between each iteration. It would do this just fine in the beginning, but then throughput would drop off.

This is interesting, because the client would never actually wait for the responses. It would just continue to issue requests at a given rate. Throughput was observed through the console.

The latency and throughput remained spiky as IO from compaction, etc. was happening.

There could be other causes, like JVM GC, scheduling contention between the compaction process and the front end process (though, that one would be a bug), etc. The data I had at the time sure seemed to show receiving traffic at a lower rate in response to what should be background IO.