[3.1.2 backport] SGW 3.1.1 using 50-75% more memory compared to 3.1.0

While testing SGW 3.1.1 on capella, I noticed some differences in memory and CPU compared to 3.1.0, as can be seen in CBG-3272. 3.1.1 uses 50-75% more memory and 30% less CPU.

The memory regressions are also present for on-prem runs, but there are no significant differences for CPU.

Let us take some write tests:

• 3.1.0: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/15998/console. Artifacts: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/15998/artifact/
• 3.1.1: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/16611/console. Artifacts: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/16611/artifact/
• Cbmonitor graph: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_310-592_run_test_f507&snapshot=hebe_311-19_run_test_e489
• We can see a 70% increase in memory usage: hhttp://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_310-592_run_test_f507&snapshot=hebe_311-19_run_test_e489#ea232aa23dc9bc563c7feb6f02df365d
• There is almost no difference in cpu utilization: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_310-592_run_test_f507&snapshot=hebe_311-19_run_test_e489#034ba0a7d72a19ba5b487943020edc6c
• For additional reference: the 3.2.0 run for this test: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/16783/console looks almost identical to the one for 3.1.1, the main difference being 3.2.0 using about 10% more CPU: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_311-19_run_test_e489&snapshot=hebe_320-86_run_test_fb78
• Looking at the following profiles:
  • 3.1.0: [perf.jenkins.couchbase.com/job/syncgteway-hebe-new/15998/artifact/172.23.100.205_syncgateway_sg_heap_230428012840_e067eb.pprof]
  • 3.1.1: [perf.jenkins.couchbase.com/job/syncgteway-hebe-new/16611/artifact/172.23.100.205_syncgateway_sg_heap_230706010742_848218.pprof]
• We can see that there is a huge difference in the memory used by "buffio.NewReaderSize": 3.1.0 uses 320.51MB, while 3.1.1 uses 800MB. There is nothing else that shows such a large difference

Similarly for read:

• 3.1.0: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/15887/console. Artifacts: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/15887/artifact/
• 3.1.1: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/16698/console. Artifacts: http://perf.jenkins.couchbase.com/job/syncgteway-hebe-new/16698/artifact/
• Cbmonitor graph: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_310-587_run_test_1859&snapshot=hebe_311-20_run_test_d74d
• Here, memory usage increases by 75% for 3.1.1.

And blackholepuller:

• 3.1.0: http://perf.jenkins.couchbase.com/job/sg_hebe_sgreplicate_multicluster/2003/console. Artifacts: http://perf.jenkins.couchbase.com/job/sg_hebe_sgreplicate_multicluster/2003/artifact/
• 3.1.1: http://perf.jenkins.couchbase.com/job/sg_hebe_sgreplicate_multicluster/2255/console. Artifacts: http://perf.jenkins.couchbase.com/job/sg_hebe_sgreplicate_multicluster/2255/artifact/.
• Cbmonitor graph: [cbmonitor.sc.couchbase.com/reports/html/?snapshot=hebe_310-592_run_bp_test_dbd1&snapshot=hebe_311-19_run_bp_test_aafe]
• Here, memory usage also increases by 75% for 3.1.1