query throughput lower due to change in ns_server affecting indexer

Avg. Query Throughput (queries/sec), Q3 Array Indexing, Covering, Range Scan, Plasma, request_plus
6.5.0-4960 - 740

7.0.0:

1165: http://perf.jenkins.couchbase.com/job/iris/17747/ - 479
http://perf.jenkins.couchbase.com/job/iris/17836/ - 479.0

1140: http://perf.jenkins.couchbase.com/job/iris/17840/ - 480.0
1120: http://perf.jenkins.couchbase.com/job/iris/17839/ - 480.0
1110: http://perf.jenkins.couchbase.com/job/iris/17842/ - 481.0
1107: http://perf.jenkins.couchbase.com/job/iris/17844/ - 481.0
1107-2: http://perf.jenkins.couchbase.com/job/iris/17846/ - 481.0 . <----

1106-2: http://perf.jenkins.couchbase.com/job/iris/17847/ - 742.0 . <----
1106: http://perf.jenkins.couchbase.com/job/iris/17845/ - 740.0
1105: http://perf.jenkins.couchbase.com/job/iris/17843/ - 745.0
1100: http://perf.jenkins.couchbase.com/job/iris/17838/ - 734.0

We are seeing roughly 25% reduction in query throughput in build 7.0.0-1107 compared to 7.0.0-1106. You can see the comparison graphs here: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_700-1107_access_cab4&snapshot=iris_700-1106_access_249f

Logs:

1107 - 45 is indexer node, 55 is n1ql node

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17846/172.23.100.45.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17846/172.23.100.55.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17846/172.23.100.70.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17846/172.23.100.71.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17846/172.23.100.72.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17846/172.23.100.73.zip

1106 - 45 is indexer node, 55 is n1ql node

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17847/172.23.100.45.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17847/172.23.100.55.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17847/172.23.100.70.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17847/172.23.100.71.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17847/172.23.100.72.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-17847/172.23.100.73.zip

Changelog:

http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=7.0.0-1106&tob=7.0.0-1107

There are only 2 relevant changes, one comes from asterixdb and the other from ns_server. The ns_server change is only adding compatibility mode markers and relevant functions. It seems this compatibility mode value is not used anywhere by n1ql but is used by indexer. If you look in the logs ns_server.indexer.log for 1107 you will see some errors that are not in the 1106 logs. Other than that I do see some connection closed type messages that could be related.