10x increase in max read/update latency with beta 2

We recently updated the YCSB durability tests from Java SDK 3 beta 1 to Java SDK beta 2. We see a slight improvement in some of the throughputs comparing server build 4908 to 4926 but what really stands out is the 20x increase in MAX read and update latency. Build 4926 is the first server build run with Java SDK 3 beta 2. To confirm that it is a SDK issue and not a server issue, I reran server build 4908 with Java SDK 3 beta 2 and the same increase is seen. Furthermore, the increase is not see in the corresponding tests for Java SDK 2 for either server build 4908 or 4926.

The data for this ticket comes from the durability SLO tests here: http://showfast.sc.couchbase.com/#/timeline/Linux/kv/durability/SLO

4908 with Java SDK 3 beta 1:

49.092 - http://perf.jenkins.couchbase.com/job/hercules/464/

http://perf.jenkins.couchbase.com/job/hercules/464/artifact/YCSB/

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.121.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.122.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.123.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.124.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.125.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.126.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.127.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-464/172.23.100.128.zip

4908 with Java SDK 3 beta 2:

1,632.186 - http://perf.jenkins.couchbase.com/job/hercules/663/

http://perf.jenkins.couchbase.com/job/hercules/663/artifact/YCSB/

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.121.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.122.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.123.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.124.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.125.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.126.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.127.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-663/172.23.100.128.zip

4926 with Java SDK 3 beta 2:

1,675.253 - http://perf.jenkins.couchbase.com/job/hercules/623/

http://perf.jenkins.couchbase.com/job/hercules/623/artifact/YCSB/

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.121.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.122.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.123.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.124.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.125.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.126.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.127.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hercules-623/172.23.100.128.zip

Comparison of 4908 beta 1 and beta 2

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hercules_650-4908_access_236b&snapshot=hercules_650-4908_access_0697

Only thing really noticeable is that average disk commit time and disk write queue in beta 2 is around double that of beta 1.