multiple regressions from same build 7.0.0-1023

There are multiple query tests and a single dcp test showing a range of regressions coming in build 7.0.0-1023.

Avg Throughput (queries/sec), Workload E, MOI, SSL, 4 nodes
6.5.0-4960 - 41229

1023: http://perf.jenkins.couchbase.com/job/hebe/5756/ - 36387
1023-2: http://perf.jenkins.couchbase.com/job/hebe/5768/ - 36494
1022-2: http://perf.jenkins.couchbase.com/job/hebe/5767/ - 40537
1022: http://perf.jenkins.couchbase.com/job/hebe/5752/ - 40375

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_700-1023_access_4226&snapshot=hebe_700-1022_access_ad11

1022 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.190.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.191.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.192.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.193.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.204.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.206.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5767/172.23.100.207.zip

1023 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.190.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.191.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.192.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.193.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.204.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.206.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5768/172.23.100.207.zip

Avg Throughput (ops/sec), Workload A, 3 nodes, 12 vCPU, replicateTo=1
6.5.0-4960 - 135377

1023: http://perf.jenkins.couchbase.com/job/hebe/5769/ - 55594
1022: http://perf.jenkins.couchbase.com/job/hebe/5772/ - 115300

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_700-1023_access_5271&snapshot=hebe_700-1022_access_67b8

1022 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.190.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.191.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.192.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.193.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.204.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.206.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5772/172.23.100.207.zip

1023 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.190.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.191.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.192.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.193.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.204.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.206.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5769/172.23.100.207.zip

Avg Throughput (ops/sec), Workload A, 3 nodes, 12 vCPU, Ephemeral, no replication
6.5.0-4960 - 838377

1023: http://perf.jenkins.couchbase.com/job/hebe/5766/ - 699787
1022: http://perf.jenkins.couchbase.com/job/hebe/5770/ - 815369

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_700-1023_access_1285&snapshot=hebe_700-1022_access_51fe

1022 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.190.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.191.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.192.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.193.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.204.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.206.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5770/172.23.100.207.zip

1023 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.190.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.191.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.192.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.193.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.204.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.206.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-5766/172.23.100.207.zip

DCP Throughput, 1 node, 250M x 1KB, Java client, compression enabled
6.5.0-4960 - 166065

1023: http://perf.jenkins.couchbase.com/job/ares-dcp/676/ - 135626
1022: http://perf.jenkins.couchbase.com/job/ares-dcp/677/ - 167416

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=ares_700-1023_access_df26&snapshot=ares_700-1022_access_c2a1

1022 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-677/172.23.133.11.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-677/172.23.133.12.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-677/172.23.133.13.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-677/172.23.133.14.zip

1023 Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-676/172.23.133.11.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-676/172.23.133.12.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-676/172.23.133.13.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-ares-dcp-676/172.23.133.14.zip

 

 

Changelog:

http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=7.0.0-1022&tob=7.0.0-1023

This commit seem like it might be causing some of the issues due to changes in SSL library: https://github.com/couchbase/kv_engine/commit/c3bf6e979b94a3d50d580ea7fc7c592e59a26438 

Not much to see in the graphs other than the expected lower ops and higher latencies. There are several commits from indexer, but no clear culprit. As for the kv engine commit, since it deals with SSL, it may be the cause for two of the regressions but not all of them. There is probably another commit that is causing the issues.