GC in query engine causes 30-40% drop in throughput and CPU utilization

Although this issue is rather obvious and well-known, we still don't have a clear baseline and an explicit goal.

Scenario: Q2, not_bounded, MOI.

According to ShowFast, Q2 throughput is 24-25K and CPU utilization is 65-70%. That definitely looks sub-optimal.

I ran a silly experiment with the following settings:

systemctl set-environment GOGC=10000

RSS and GC pauses increase to absolutely impractical values. However, throughput periodically increases up to 40-42K and CPU utilization increases up to 100%.

Note, I ran a whole bunch of other experiments (pretty=false, metrics=false, GOMAXPROCS=2 x numCPU, diff. service deployments, Go 1.6, Go 1.7). Nothing helps to improve overall throughput and CPU utilization.

Also both pprof and perf indicate significant issues with GC.

I think we should shoot for 40K queries/sec and 90% CPU utilization.