Currently the only way to tell Prometheus about your cluster (unless you're in k8s) is to use file service discovery. This isn't a trivial process for people unfamiliar with containers.
I created a PoC of using cbmultimanager's cluster list as the source of truth for Prometheus, and it's promising.
- Do we keep around cbmm's existing Prometheus discovery?
- IMO yes, customers may want to use Prometheus' other SD mechanisms and we can't support them all
- If ^, what happens if you create a SD cycle?
- No problem, Prometheus handles the same cluster being exposed both by file and by HTTP without any issues.
- How do we handle OSS builds? (Not including cbmm will mean Prometheus tries to HTTP-SD a nonexisting port - it'll still function, but logs will be flooded with errors, which is not ideal)