Ok, closing it out.  It turned into quite a sprawling ticket with various ideas and testing of performance improvements, but the core takeaway is that transactions performance is reasonably close to the maximum currently possible, considering durability performance and the current protocol.  There is at least two ways the protocol can be improved in the future, performance-wise, which have been logged separately.

Graham Pople - I guess we can close this ticket .

Hi Sharath Sulochana , do you think we still need this ticket open?

Shivani Gupta it's the latter - the ATRs are created on-demand.

Admittedly, the numbers above do indicate that the number of ATRs isn't much of a factor when Durability is enabled.  Though countering that is issue MB-35359, where transactions were actually expiring due to congestion on the ATRs.  It's been worked around, as newer Java clients now automatically retry on that error for up to 2.5 seconds, so the issue is closed - but it remains indicative that there's heavy congestion going on behind the scenes.

Graham Pople question for you:

Today do you create the 1024 ATRs upfront or the first time a transaction hits its first document on a given vbucket?

Because when I run small experimental tests I do not see 1024 ATRs right away (which is a relief actually!).

I agree with your concern that if we go with 20 *1024 ATRs, they will flood the users bucket. Until we have a way of filtering out ATRs into a special System Collection I would not create tons of them.

Also, given that with Durability level = majority, the bottleneck is elsewhere rather on ATR contention, I am less convinced this change is necessary.