Currently, our throughput tests are measuring how many client requests the cluster can handle, and I think the number is important. The same concept is applied to latency tests. We measure how fast client requests can be completed. The latency is measured when the requests are completed but not when replication is done. I agreed DCP replication performance is important, but we should measure the two things separately. 

1. client request process performance (the throughput we measure)
2. dcp replication performance

If the client request process performance is 10% slower and DCP replication performance is 20% faster, I don't think it means the overall system performance is better.

Rebalance tests are one method we use to measure DCP replication performance. If there are any ideas about how we should measure DCP replication performance, please let me know. For this particular test (insert only), another thing we can try is to do a run without replication. 

We ideally want to measure the sustainable write throughput (which includes replication not lagging). This will require the test framework to slow down and adjust the requests rate sufficient enough to balance replication. Maybe reducing the number of clients may slightly help here. If we wait for replication to complete for the measurement, the throughput number itself will not be relevant, but it can be used as a number for catching regressions.
Since this is a sanity test for the purpose of specially measuring storage engine insert rate, changing the test to no replica config also should be okay.

I agree with both of you - we want to see how fast KV-Engine can process client requests (and under what latency), but we also want to measure how fast KV-Engine can handle the full lifetime of a request (processing off network, writing to local disk; replicating to remote cluster and writing to disk there).

As Bo-Chun suggested, the first type of test doesn't really care about replication; so we could simply run without replicas; measuring the throughput where we just process the request on the active node and write it to disk.

The second type of test is pretty much the test we have here (I believe); however we do want to make sure the performance of the system can be sustained; and we are not building up backlogs which only recover when the workload stops running - i.e. if the test ran for a longer period we would see a drop in throughput long-term.

Note that KV-Engine will already apply backpressure to the client(s) if persistence / replication cannot keep up (and Bucket memory has been saturated). This back pressure should be better behaved in Neo due to the changes done by Paolo Cocchi and James Harrison recently - namely giving outstanding mutations to be written to disk / replicated (CheckpointManager) their own sub-quota of the overall Bucket quota. We are still tuning this, but currently the CheckpointManager gets a maximum of 30% of the overall Bucket quota in Neo; whereas previously it could in theory consume the entire Bucket quota (leaving essentially zero memory for caching recently accessed items / disk backfills for other DCP clients etc).
I mention this because if we have perf tests which KV-Engine cannot sustain the operation rate, but for the (short) duration they run there was sufficient Bucket Quota in 7.0.0 to cache dirty / queued for replication items, we could observe lower throughput under Neo as we would start applying back-pressure to clients "sooner". I would argue that this isn't really a regression, the test simply didn't run for long enough in the first place as a similar drop in throughput would be seen in 7.0.0, if the test duration was extended.

Closing all Duplicates, Not a Bug, Incomplete, Duplicate