Query throughput decreased using plasma

We are observing a decrease in query throughput for many different benchmarks, all using Plasma. 

The following tests all have similar issues:

• Avg. Query Throughput (queries/sec). CI19. Composite Filters (1000 -> 1). Covered. Plasma. request_plus
• Avg. Query Throughput (queries/sec). Q19. Composite Filters (1000 -> 1). Plasma. request_plus
• Avg. Query Throughput (queries/sec). CI19. Composite Filters (1000 -> 1). Covered. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). Q19. Composite Filters (1000 -> 1). Plasma. not_bounded
• Avg. Query Throughput (queries/sec). AG6. Aggregation query. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). AG1. Aggregation query. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). AG5. Aggregation query. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). AG9. Aggregation query. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). Q20. Count (10K matches). 100M docs. Plasma. request_plus
• Avg. Query Throughput (queries/sec). Q20. Count (1K matches). 100M docs. Plasma. request_plus
• Avg. Query Throughput (queries/sec). Q20. Count (1K matches). 100M docs. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). Q2. Singleton Unique Lookup. Plasma. DGM. request_plus
• Avg. Query Throughput (queries/sec). Q2. Singleton Unique Lookup. Plasma. DGM. not_bounded
• 90th percentile query latency (ms). Q2. Singleton Unique Lookup. 450 queries/sec. Plasma. request_plus
• Avg. Query Throughput (queries/sec). Q18. Range Scan with Limit and Offset (1K). Plasma. request_plus
• Avg. Query Throughput (queries/sec). Q18. Range Scan with Limit and Offset (1K). Plasma. not_bounded
• Avg. Query Throughput (queries/sec). CI18. Range Scan with Limit and Offset (1K). Covering. Plasma. not_bounded
• Avg. Query Throughput (queries/sec). CI18. Range Scan with Limit and Offset (1K). Covering. Plasma. request_plus

I will only provide data for one of the tests. If more data is require please let me know.

Test

Avg. Query Throughput (queries/sec). Q18. Range Scan with Limit and Offset (1K). Plasma. not_bounded

Index

CREATE INDEX by_capped_small ON `bucket-1`(capped_small);

Query

SELECT name, email FROM `bucket-1` WHERE capped_small > $1 ORDER BY capped_small OFFSET 1000 LIMIT 10;

Results

6.5.0-3123: 20338.0 q/s

6.5.0-3124: 11416.0 q/s

Report

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_650-3123_access_6926&snapshot=iris_650-3124_access_b6b4

Logs

6.5.0-3123

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14584/172.23.100.45.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14584/172.23.100.55.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14584/172.23.100.70.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14584/172.23.100.71.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14584/172.23.100.72.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14584/172.23.100.73.zip

6.5.0-3124

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14585/172.23.100.45.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14585/172.23.100.55.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14585/172.23.100.70.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14585/172.23.100.71.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14585/172.23.100.72.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-14585/172.23.100.73.zip

Changelog

http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=6.5.0-3123&tob=6.5.0-3124

PProf

6.5.0-3123

http://perf.jenkins.couchbase.com/job/iris/14584/artifact/172.23.100.55_n1ql_cpu_697116.pprof

6.5.0-3124

http://perf.jenkins.couchbase.com/job/iris/14585/artifact/172.23.100.55_n1ql_cpu_490cc3.pprof

Comment

I couldn't tell much from the pprof graphs, but if you look at the changelog it seems like the issue might be coming from some plasma commits.