KV: Scalable Stats

Scalable Stats PRD

From the spec for Stats in Cheshire Cat

With the introduction of collections, the number of stats generating entities in the system will grow dramatically. In particular, we will have 1000s of collections and many 1000s of global secondary indexes.

In Mad-Hatter and prior releases the volume of stats collected by ns_server is normally fairly modest and this allows ns_server to collect stats every second, continually roll them up across different time granularities and store all stats in memory and use non-optimized representation on disk. Additionally, stats don't greatly affect the size and upload time associated of cbcollects.

With the move to collections, the number of individual metrics the system will need to track will increase by approx 2 orders of magnitude. Necessarily, the way we collect, store, serve and upload stats needs to be completely rethought.

To cope with a much larger number of stats, ns_server is moving to use Prometheus to collect and store stats efficiently. To support this, KV needs to integrate an exporter capable of generating the Prometheus Text Format and serving it over HTTP.

KV is required to expose two endpoints: /_prometheusMetrics for per-bucket level metrics and /_prometheusMetricsHigh for higher cardinality metrics e.g., per-collection. These endpoints should both have HTTP Basic Auth.