Details
-
New Feature
-
Status: To Do
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
Currently the only way to tell Prometheus about your cluster (unless you're in k8s) is to use file service discovery. This isn't a trivial process for people unfamiliar with containers.
I created a PoC of using cbmultimanager's cluster list as the source of truth for Prometheus, and it's promising.
Outstanding questions:
- Do we keep around cbmm's existing Prometheus discovery?
- IMO yes, customers may want to use Prometheus' other SD mechanisms and we can't support them all
- If ^, what happens if you create a SD cycle?
- No problem, Prometheus handles the same cluster being exposed both by file and by HTTP without any issues.
- How do we handle OSS builds? (Not including cbmm will mean Prometheus tries to HTTP-SD a nonexisting port - it'll still function, but logs will be flooded with errors, which is not ideal)
My preference here would be we provide a configuration service in the microlith that both Prometheus and cluster monitor can use. This resolves the OSS/enterprise split as they can both use it plus we can provide a focussed solution to cope with configuration issues directly rather than tacking more requirements on to cluster monitor.
Users can also pick it up as well as another endpoint.