Holistic way to suppress an alert/check

Our alerts are best practices for production environments, but there will be many cases where there are reasons for the end user to deviate from them. We should have a way of telling CMOS that a failing check is expected and suppress the alert.

Alertmanager already has silences, but a silenced alert will still appear in the dashboards etc (we could possibly just replace all our dashboards with Alertmanager data, though this would need careful consideration). Alternatively, Cluster Monitor has a suppression system that we may be able to take advantage of.