I think to a large degree this is expected, as many of the underlying operations implementing transactions are not cmd_get/cmd_set, but rather using the subdocument API with xattrs.

Sharath Sulochana: do you have the full set of (non zero) KV statistics?

Matt Ingenthron -  Thanks for looking into the ticket . I had this assigned to me as I was still planning to add some more details . 

here is a link to one of the kv test as you requested - 

http://perf.jenkins.couchbase.com/job/hebe-dura-txn/227/console

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_650-3883_access_a4b1

I will share additional details for transaction test with same config . 

100% of txn operations are done with Sub-Document, so I suspect those cmd_gets and cmd_sets do include the Sub-Document ops.

Thanks for these early figures Sharath.

They do look on the low side, both in terms of the reduced overall number of operations, and the transactions/sec which is lower than I'd expect with those ops/secs figures - but it's not completely unexpected.  This initial release of transactions is focussed on stability and as yet hasn't been subject to excessive profiling.  There's doubtless some low hanging fruit, and I look forward to having time after the beta release to investigate and improve this area.  I'll be surprised if we don't see some big improvements to these numbers in the coming weeks.

In particular transactions is mostly just doing KV ops, so the high CPU usage is very interesting.  I suspect the logging needs some iteration.

Note from the performance meeting on August 7, the CPU usage concern is actually at the server side. Sharath Sulochana is going to get more information into the ticket, as it was raised that CPU was a larger concern, but that didn't come through in the initial description.

It may be worth microbenchmarking the subdoc operations being used at some point.

Sharath Sulochana: The CPU utilization in the description, is that Irix style accounting (where 16 cores is 1600%) or Solaris style accounting (where use of 4 of 16 cores would be 25%)? This came up as a point of confusion in another perf test not too long ago.

Matt Ingenthron

CPU captured in perfrunner are  based on ns_server stats api via _+

:{port}/pools/default+_  . 

I believe it uses top command internally (where 16 cores is 1600% and then averages it )  .  Need to confirm it with ns_server team .

Thanks. If that's the case, then going from 38% to 80% doesn't seem unreasonable as subdocument ops are a little more expensive and there are 4 times as many.

It looks like to know where to take this next, it'll be great when you can pass along the set of stats for KV and how CPU is being accounted for Sharath Sulochana.

Along with the description of the SUT as well please Sharath Sulochana. How many CPUs, etc.

Matt Ingenthron : I dig little deeper on these tests . Looks like I had compared two wrong tests . Basically we have three set of tests current YCSB KV, new durability & transactions tests .  I had inherited configs from KV tests (with lower cores ) . I will be closing this as not a bug . 

Reopening it after reassessing it based on new tests results .

The numbers do point towards some bottlenecks, a few theories on where they are:

1. ATR contention.  There are 1024 ATRs and each transaction requires 4 writes to one of them.  8k txns/sec are going to generate 32k ops/sec on just 1024 docs, and while there should be no contention on the content itself (as they're writing to different paths in the doc), under-the-hood there's a great deal of document contention.   With Durability=None the contention is resolved quickly server-side in a CAS loop (fetch the full doc, apply the subdoc mutations, try to write the doc, retry on CAS fail), otherwise it is resolved by DurabilitySyncWriteInProgress errors being sent back to client and the client retrying.

I'm very curious what will happen if we change the number of ATRs, and I'm adding TXNJ-112 to let YCSB configure it. There will be a linear increase in required for the backround cleanup, which polls each ATR every minute, but it's pretty minimal (17 reads/sec currently, so can easily increase by 10x).  Once that's in Sharath will run tests with various numbers of ATRs (suggest 1024*1,5,10 & 20) and we'll see what drops out.

2. High cluster CPU (90%+).  Though I'm making small changes to the ATRs using subdoc, with durability the entire ATR doc is sent in the DCP Prepare.  I wonder if these docs are getting pretty big, and the server is spending a bunch of time a) reading the full doc to apply the subdoc change, (which is always going to need to happen) and b) parsing large DCP Prepares on the replicas (when theoretically instead the Prepare could contain just the subdoc mutation rather than the full doc - though I briefly chatted with KV about this and get the impression it's non-trivial).  This perhaps accounts for the high CPU seen on the cluster.

I'm not sure changing the number of ATRs will have any impact here - it will be processing more, smaller docs, but overall the same amount of data.

Under TXNJ-110 I'll add summary diagnostic events on the size of ATRs, which hopefully Sharath can also integrate into YCSB.  This may give us something to go on, though this sub-issue is probably better investigated by KV team.

3. Client-side CPU & GC churn.  Sharath is going to add client-side CPU monitoring.  This is a new library, it hasn't gone through profiling yet, it logs heavily, and it's entirely possible there's some low-hanging fruit to address.

Bonus. Poor YCSB distribution.  I've seen evidence, both in my transactions logging and in pcaps, that though YCSB is spinning up many clients, only a handful (possibly just 1 or 2) in each worker are doing any real work, which will likely impact throughput.  This is somewhat contended, as Sharath has investigated and believes YCSB is distributing the workload just fine.  Nonetheless, I'd like to find time to spin up YCSB locally and investigate this further.

4. Unexpected reads.  While debugging with Sharath we also noted that there should be around 32k reads/sec (~8k txns/sec, each doing 4 reads), but we're seeing ~37k.  That missing 5k needs to be accounted for, possibly using some advanced wireshark-fu.

This shouldn't have much performance impact though, as in this test the cluster was capable of ~320k ops/sec.  I'm just curious about it.

Focussing on 1. ATR Contention here.

Under TXNJ-112 I made the number of ATRs a configuration option and have been experimenting with it today, with promising results from YCSB:

So we see a big initial leap in performance going from the default 1024 ATRs up to 2048 (10% improvement) then 4096 (22%), and then a steady improvement all the way up to the current configurable max of 20,480 ATRs (40% improvement). 

Performance setup: 3 AWS m4.large nodes; 100 YCSB threads; Thinkpad Carbon X1 laptop; 200k records; durability=1; transactions beta.2-SNAPSHOT; run for 5 minutes each
Other results: server CPU pegged at ~95% even from initial 1024 test; client CPU varied 45-65%; ops ~800*3 at 1024 ATRs, ~950-1350*3 (quite peaky) at 20480

MB-35649 contains an important performance fix for sync rep that is going out in 4133, and will likely have an impact on txns performance too.

Sharath Sulochana, I'm drilling into the YCSB code and the couchbase3-transactions branch is missing some crucial changes that Michael Nitschinger made in couchbase3-new-durability branch.  Specifically this will share one Java client ClusterEnvironment between all threads (Couchbase3Clients really) in the ycsb process, leading to much more efficient use of resources.

Can you please merge these changes in?  Note that it will be imperative to set the kvEndpoints parameter after.  I'd suggest setting it to double the number of threads.  That will let each Couchbase3Client have its own kvEndpoint effectively, plus some wiggle room.

also Korrigan Clark is able to provide input on the params he is using for the durability testing Sharath Sulochana so it makes sense for both of you to sync up on this.

Graham Pople - Here is the numbers form latest tests with numATR's set to 20480 (build - 6.5.0-4081) .  There is some significant increase in throughput when durability level is NONE (almost 100%+) . But other durability levels have improvement at ~10% .

Note - Thanks for kvEndpoints suggestion . I will go ahead and implement the changes  .... 

Thanks Sharath Sulochana for the latest testing.  Don't suppose you have the figures saved from testing with other values of numAtrs?  I want to increase it but will need to make a judgement call balancing the best performance improvement vs "woah this seems like a lot of metadata documents to create in the customer's cluster".   Also it will guide me on whether it's worth increasing the max limit even beyond the current 20k-ish.

Good to see the big performance jump, at least with durability disabled.  For Majority+ I suspect we've hit some other bottleneck that is preventing the numATRs change from shining, and my guess would be the high cluster-side CPU.  Daniel Owen can we possibly request a bit of time from your team on profiling the CPU usage (presuming it's from kvengine, Sharath Sulochana can you confirm)?  Transactions does some unusual things (lots of churn in big xattrs, for example), so maybe there's some low-hanging performance fruit in KV here?  Or perhaps we'll need to take another approach for txns entirely - either way, would be very helpful to get some expert help at this point.

Hi Graham Pople. Yeh that fine. Probably easiest is when the performance team does a run, for them to use linux perf to capture a profile for us to analyse.

Though the transaction throughput figures look low, it's important to remember how much each transaction is doing: not only updating multiple documents, but also updating the ATRs so the txn can be cleaned up if anything goes wrong.  I'm crunching the numbers, and for Majority+, the transactions throughput is actually very close to the current theoretical maximum *.

Each transaction tested is doing 14 operations total (4 reads, 4 writes to the ATR, and 3 updates which involve a stage and a commit each, so 6 writes there).  So e.g. at Majority we have 1,394 txns/sec, which is 1,394 * 14 = 19,516 operations per sec - within spitting distance of the regular KV throughput you've achieved of 21,315 ops/sec. 

Here's the full table:

 
So, we're actually doing pretty well.

(* Note the current theoretical maximum can be improved slightly, as we can probably get rid of 1 of one of the ATR writes.)

Graham Pople question for you:

Today do you create the 1024 ATRs upfront or the first time a transaction hits its first document on a given vbucket?

Because when I run small experimental tests I do not see 1024 ATRs right away (which is a relief actually!).

I agree with your concern that if we go with 20 *1024 ATRs, they will flood the users bucket. Until we have a way of filtering out ATRs into a special System Collection I would not create tons of them.

Also, given that with Durability level = majority, the bottleneck is elsewhere rather on ATR contention, I am less convinced this change is necessary.

Shivani Gupta it's the latter - the ATRs are created on-demand.

Admittedly, the numbers above do indicate that the number of ATRs isn't much of a factor when Durability is enabled.  Though countering that is issue MB-35359, where transactions were actually expiring due to congestion on the ATRs.  It's been worked around, as newer Java clients now automatically retry on that error for up to 2.5 seconds, so the issue is closed - but it remains indicative that there's heavy congestion going on behind the scenes.

Hi Sharath Sulochana , do you think we still need this ticket open?

Graham Pople - I guess we can close this ticket .

Ok, closing it out.  It turned into quite a sprawling ticket with various ideas and testing of performance improvements, but the core takeaway is that transactions performance is reasonably close to the maximum currently possible, considering durability performance and the current protocol.  There is at least two ways the protocol can be improved in the future, performance-wise, which have been logged separately.