replicate_to=1, persist_to=0 inconsistent behaviour

I have been running some experiments with the SDK 3.0 and have seen some odd behaviour relating to performance for replicate_to=0 + persist_to=0.

In the Threads tab in this sheet: https://docs.google.com/spreadsheets/d/1B8v4OZneOeGxJwUj226zA3YDr0Y0gjRSVLwy0IAP9qw/edit?usp=sharing

You can see that replicate_to=1 + persist_to=0 out performs replicate_to=1 + persist_to=1 in the single client machine + single thread scenario only. When client machines are scaled up and when client threads are scaled up, replicate_to=1 + persist_to=1 out performs replicate_to=1 + persist_to=0 consistently.  In the Throughout tab we can see then that replicate_to=1 + persist_to=0 performs worse than replicate_to=1 + persist_to=1 and replicate_to=1 + persist_to=2. It appears as though we take a hit for NOT persisting and then a smaller hit for each replica we persist to after the first. The expected performance hierarchy should be the following: replicate_to=1 + persist_to=0 > replicate_to=1 + persist_to=1 > replicate_to=1 + persist_to=2, where > means more throughput. This issue should not be related to SDK and Michael Nitschinger have been working to eliminate that possibility. In addition to the odd performance hierarchy, in the Throughput tab you can also see that replicate_to=1 + persist_to=0 using SDK2 out performs replicate_to=1 + persist_to=0 using SDK3 by 18x.