Running additional tests to narrow down the potential root cause.

Looks like the regression was introduced in 7.1.0-2473.

http://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=7.1.0&fromBuild=2472&toVersion=7.1.0&toBuild=2473&f_backup=off&f_build=off&f_cbas-core=off&f_cbft=off&f_eventing=off&f_indexing=off&f_product-metadata=off&f_query=on&f_testrunner=off&f_asterixdb=off&f_kv_engine=off&f_magma=off&f_ns_server=off&f_platform=off&f_query-ui=off

Backing out MB-51372 and the figure is 664 (http://perf.jenkins.couchbase.com/job/arke/10253) for "90th percentile query latency (ms), Q26, Hash Join (1K -> 100K), not_bounded" which isn't quite back to 632 but still indicates the issue.

But we can't not have this fix as it prevents a crash.

The overhead comes from unavoidable additional allocations per covered value - and these tests rely on covered values; the tests have been falsely fast previously because of the bug.

I will try another idea out that might be able to avoid the crash and the allocations and will update again later.

http://perf.jenkins.couchbase.com/job/arke/10254 - 656  # type

http://perf.jenkins.couchbase.com/job/arke/10255 - 643  # break in fns

http://perf.jenkins.couchbase.com/job/arke/10256 - 663  # copiedObject

http://perf.jenkins.couchbase.com/job/arke/10257 - 913  # copy

http://perf.jenkins.couchbase.com/job/arke/10259 - 659  # detect fields

http://perf.jenkins.couchbase.com/job/arke/10260 - 653  # detect flag

http://perf.jenkins.couchbase.com/job/arke/10261 - 734  # use NULL_VALUE

http://perf.jenkins.couchbase.com/job/arke/10262 - 656  # copiedObject 2nd run

http://perf.jenkins.couchbase.com/job/arke/10264 - 649  # aliased type

Plain 2475 run - 785 (again)

Build couchbase-server-7.1.0-2508 contains query commit 19e7ac8 with commit message:
MB-51492 Revise fix.

There will be a small increase in latency with this revised fix in place, but it should be much closer to the base line.

Build couchbase-server-7.2.0-1020 contains query commit 19e7ac8 with commit message:
MB-51492 Revise fix.

Verified with build 7.1.0-2508.

The performance regression has been fixed.