Analytics incr. ingestion 60% slower on 7.6 vs 7.2.4 with default collection

For 7.6.0 release we ran our "P1" performance tests on all components. These tests don't get ran every week like "P0" tests. We observe 60% lower incremental ingestion rate with 7.6 vs 7.2.4 in the following P1 test that uses the default collection:

Avg. incremental ingestion rate (items/sec), 4 nodes, BigFUN 20M users (320M docs), 3 indexes, SSD

http://showfast.sc.couchbase.com/#/timeline/Linux/analytics/sync/Incremental#bigfun_20M_sync_4n_incr_oceanus

In our other incremental ingestion tests (see them all here), 7.6.0 is only ~20% worse than 7.2.4, so this seems significant.

I know 7.5 is in the middle there but I'm not sure its a good baseline to compare to because 7.5 also did very poorly in our ingestion tests with lots of collections (like this one), whereas 7.2.4 behaves more similarly to 7.6.0 in all other tests. That being said, understanding why 7.5.0 has half the ingestion rate of 7.2.4 might be the first step in understanding what is going on.

Since this was the first time we've ran this test on 7.6.0, I ran with an older 7.6.0 build to start understanding when the performance dropped.

When looking into analytics incremental ingestion, I discovered quite a saga of tickets and efforts to improve ingestion rate when there are lots of collections, which has sometimes been to the detriment of lower collection-count tests. We need to establish whether this performance is expected or not.