Performance Regression: Durability Majority Persist Active

We are seeing a significant decrease in throughput for YCSB workload A with durability level Majority Persist Active. Throughput decreased around 50% for SDK3 alpha7 and around 33% for SDK3 alpha6. Note, there is a difference in throughput between Alpha6 and Alpha7 due to new retry strategy in Alpha7, but there is a separate issue for that: https://issues.couchbase.com/browse/JCBC-1447

SDK3 Alpha6:

6.5.0-4131 http://perf.jenkins.couchbase.com/job/hebe-dura-txn/444/ - 9150 
6.5.0-4132 http://perf.jenkins.couchbase.com/job/hebe-dura-txn/447/ - 7706

6.5.0-4131:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.190.zip
 https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.191.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.192.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.193.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-444/172.23.100.207.zip

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_650-4131_access_23ea

http://perf.jenkins.couchbase.com/job/hebe-dura-txn/444/artifact/YCSB/

6.5.0-4132

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.190.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.191.zip
 https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.192.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.193.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-447/172.23.100.207.zip

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_650-4132_access_d3a4

http://perf.jenkins.couchbase.com/job/hebe-dura-txn/447/artifact/YCSB/

SDK3 Alpha7:

6.5.0-4131 http://perf.jenkins.couchbase.com/job/hebe-dura-txn/460/ - 6689
6.5.0-4132 http://perf.jenkins.couchbase.com/job/hebe-dura-txn/467/ - 3491

6.5.0-4131

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.190.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.191.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.192.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.193.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-460/172.23.100.207.zip

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_650-4131_access_5103

http://perf.jenkins.couchbase.com/job/hebe-dura-txn/460/artifact/YCSB/

6.5.0-4132

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.190.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.191.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.192.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.193.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-dura-txn-467/172.23.100.207.zip

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_650-4132_access_80f5

http://perf.jenkins.couchbase.com/job/hebe-dura-txn/467/artifact/YCSB/

Analysis:

Across the board, cpu utilization is both less and more volatile for 4132.

Changelog:

http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=6.5.0-4131&tob=6.5.0-4132

Only one commit here related to durability: 

https://issues.couchbase.com/browse/MB-35458
https://github.com/couchbase/kv_engine/commit/f6ed489642e80594ff6a9a87a12f3d9c343fccfc