Detect and/or workaround systemd incorrect swappiness in RHEL8 & similar era distros

While investigating CBPS-1208, we found that despite setting vm.swappiness = 0, we were getting aggressive swap behaviour (with a big performance penalty) because all cgroups had their own swappiness setting, which was 60.

There is a good RedHat article describing the problem here: https://access.redhat.com/solutions/6785021

Any system with cgroups v1 (there are two main versions - 1 and 2) is susceptible to this, and it is quite painful to encounter.

We hit this issue in perf team after upgrading our test servers from CentOS 7 to Ubuntu 20. It is reasonable to assume that customers may hit this issue in the future, and so it makes sense to investigate what we could do from a product POV to mitigate this problem.

The issue is pretty easy to diagnose (there is an example in the RH article above) and there are workarounds such as switching to cgroups v2. One quick thing we could do is update our documentation to cover this.

A more proactive solution could be to set the swappiness ourselves when running on systems with cgroups v1.