7.6.0-1816 XDCR Rate decreases in tests with multiple nodes and 1000 collections

XDCR rate decreased by 25% between 7.6.0-1782 and 7.6.0-1816, as can be seen here: http://showfast.sc.couchbase.com/#/timeline/Linux/xdcr/init_multi/all#xdcr_init_5x5_unidir_100M_1s_1000c_dgm_comp_titan_5x5

The regressions happened in DGM tests with multiple node clusters and 1000 collections.

• Avg. initial XDCR rate (items/sec), 5 -> 5 (2 source nozzles, 4 target nozzles), 1 bucket x 1G x 1KB, DGM, s=1, c=1000
  • 7.6.0-1782: https://perf.jenkins.couchbase.com/job/titan/15317/console
  • 7.6.0-1812: https://perf.jenkins.couchbase.com/job/titan/15343/console
  • Cbmonitor report comparisons: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_760-1782_init_xdcr_6b86&snapshot=titan_c1_760-1816_init_xdcr_8932 and http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c2_760-1782_init_xdcr_a53c&snapshot=titan_c2_760-1816_init_xdcr_8699.
  • In the second cbmonitor report, we can see that throughput is similar for most of the test, until close to the end, where the throughput in the run with 7.6.0-1812 drops by 80-90%.
• Avg. initial XDCR rate (items/sec), 5 -> 5 (2 source nozzles, 4 target nozzles), 1 bucket x 1G x 1KB, DGM, s=1, c=1000, external_map:
  • 7.6.0-1782: https://perf.jenkins.couchbase.com/job/titan/15320/console
  • 7.6.0-1812: https://perf.jenkins.couchbase.com/job/titan/15343/console
  • Cbmonitor report comparisons: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_760-1782_init_xdcr_eff6&snapshot=titan_c1_760-1816_init_xdcr_84fa and http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c2_760-1782_init_xdcr_6679&snapshot=titan_c2_760-1816_init_xdcr_b6e7
  • The behavior is identical to the previous test, throughput is similar for most of the test and drops off close to the end.

It is worth noting that the regression doesn't happen in equivalent tests with only 1 node, such as this: http://showfast.sc.couchbase.com/#/timeline/Linux/xdcr/init_single/all#xdcr_backfill_1x1_100M_1s_1000c_external_map_titan_2x2_5node1