Hey Patrick Stephens, can you review the requirements just updated above? Then, I would suggest we reassign or break it down with some tasks, targeting 7.x first. I know that earlier releases are also required, but maybe as a first-order task, try to get a 7.x sample, iterate on that, then work back to CNPE with possible configuration for 7.x style metrics or just use it's current approach, then PSPE.

Other approaches can work too, but what I like about this is that it gets the destination established and iterated on first, filled in by other things that will be steps there as 7.x adoption increases.

Of course, please bring up any questions to Anil Kumar.

A couple of questions I have…

• This calls out specifically healthy/unhealthy. Seems okay to start developing a dashboard with this, but per previous discussions there are some indicators that are binary, some that are subjective. Anil Kumar: do we have a definition on healthy/unhealthy? Any thoughts on how to get to that?
• It's not quite clear to me where the health check output (available in CMOS as a prometheus endpoint, but needs to be configurable and visualized somewhere) would be. Anil Kumar, any thoughts on where this would fit in your tables?
• This may be somewhere I'm not familiar with, but is there a repository of screenshots or Grafana JSON definitions somewhere already as previous examples?

Matt Ingenthron

is there a repository of screenshots or Grafana JSON definitions somewhere already as previous examples?

the linked requirements doc includes a prior-art section links to the (many) varied JSON dashboard definitions we have

It's not quite clear to me where the health check output would be

This can just feature as a regular panel on each of the hierarchical dashboards. e.g. on the "single cluster view" - a simple aggregated visualisation that conveys ("are all of the health checks passing?") - with an appropriate "details" panel that highlights individual failures. The Universe dashboard would again contain an aggregation of these by cluster. There are some nice hexagonal visualisations that neatly convey whether a node / cluster is passing all of the necessary checks.

Thanks David Haikney

I have updated the table and added a couple of rows to clarify the above questions. Thanks!

I know the requirements doc mentions all of the other exporters that are out there.  I would think initially it should be focused on 7.0 as Matt Ingenthron mentioned.  My concern with trying to make the pre-7.0 exporters work with CMOS is those exporters are more point in time metrics and not aggregate metrics like the majority of the metrics in 7.0, plus all of the metric names have changed completely.  

Additionally, I have not done complete testing with the cbprometheus_python exporters, but I know at a minimum the index metrics we were gathering using the /pools/default/buckets/@index-{BUCKET}/nodes/{NODE}:8091/stats endpoint in 6.6 and earlier, returns a "200 OK" but an empty payload in 7.0 and at this time there is no plans to add support for that exporter in 7.0.  

If we're going to add support for any of the previous exporters in CMOS, I would say we would only "officially" support the Couchbase-exporter built by engineering for CAO, just my opinion.  But I think it will be very difficult to support these older exporters as all of the PromQL queries will be completely different. 

I think this is on you to review