Major Description
No matter how clever we try to be with our alerts and metrics, every end user is different: for example, some may need a bucket to always be 100% resident, while others will be able to tolerate RRs as low as 20%. For threshold alerts there isn't a "one size fits all" solution.
We have a system for customising Prometheus rules (cf. CMOS-23), but it is limited. We have env-subst, but that requires manually setting an environment variable for every single tunable. Investigate if there are better ways of achieving this.