latency spike and wait time spike after kv commit

We are observing an increase in 99.99th percentile latency.

The following tests all have performance issues:

• 99.99th percentile GET latency (ms). 4 nodes. 1B x 1KB. 15K ops/sec (90/10 R/W). 5% cache miss rate. no compaction

• • ~220ms to ~320ms

-----------------------------------------

Test

Avg. Query Throughput (queries/sec). AG8. Aggregation query. Plasma. not_bounded

Results

3388 - http://perf.jenkins.couchbase.com/job/hera/2481/ - 307.0

3361 - http://perf.jenkins.couchbase.com/job/hera/2482/ - 290.0

3348 - http://perf.jenkins.couchbase.com/job/hera/2483/ - 321.0

3346 - http://perf.jenkins.couchbase.com/job/hera/2485/ - 332.0

3345 - http://perf.jenkins.couchbase.com/job/hera/2487/ - 199.0

3344 - http://perf.jenkins.couchbase.com/job/hera/2486/ - 237.0

3342 - http://perf.jenkins.couchbase.com/job/hera/2484/ - 221.0

Report

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_650-3345_access_e72c&snapshot=hera_650-3346_access_3542

Logs

6.5.0-3345

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2487/172.23.96.117.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2487/172.23.96.118.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2487/172.23.96.119.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2487/172.23.96.120.zip

6.5.0-3346

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2485/172.23.96.117.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2485/172.23.96.118.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2485/172.23.96.119.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hera-2485/172.23.96.120.zip

Changelog

http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=6.5.0-3345&tob=6.5.0-3346

Comment

The commits between 3345 and 3346 look like they only pertain to ephemeral bucket but this test uses couchbase bucket, so not to sure why this would affect it. I ran the tests multiple times and the behaviour looks consistent. If you look at the report, you will see that average background fetcher wait time increases significantly across all percentile levels, average disk commit time is higher for higher percentiles and latency, average get and set latency also increased notably in the higher percentiles for some nodes.