[BP - 5.5.6] 30% drop in TP for array queries

We are seeing a 30% drop in throughput for array queries.

Test: Avg Throughput (queries/sec), Workload SK, 4 node

http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/soe/all

Good Build: 6.5.0-3949, 69k q/s

http://perf.jenkins.couchbase.com/job/hebe/4123/

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.190.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.191.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.192.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.193.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4123/172.23.100.207.zip

Bad Build: 6.5.0-3950, 49k q/s

http://perf.jenkins.couchbase.com/job/hebe/4122/

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.190.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.191.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.192.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.193.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.204.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.206.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-4122/172.23.100.207.zip

Graphs:

http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_650-3949_access_af6a&snapshot=hebe_650-3950_access_c6ae

Analysis:

From the graphs it looks like average req and service time have increased between 3-6x, high cpu utilization. It looks like the regression is coming from this commit: https://github.com/couchbase/query/commit/13234e4e50a290c4aefd769f3af2d09bcc6f40cc.

Query:

SELECT RAW meta().id FROM `bucket-1` WHERE ANY v IN visited_places SATISFIES ANY c IN v.cities SATISFIES (v.country || "." || c) = $1 END END ORDER BY META().id LIMIT $2

Index:

CREATE INDEX `sk` ON `bucket-1`((distinct (array (distinct (array (((`v`.`country`) || ".") || `c`) for `c` in (`v`.`cities`) end)) for `v` in `visited_places` end))) WITH { "nodes":[ "172.23.100.204:8091","172.23.100.205:8091","172.23.100.206:8091","172.23.100.207:8091" ], "num_replica":3 }