Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.4, 7.2.0
-
None
-
Untriaged
-
0
-
Unknown
-
KV 2023-4
Description
Some of the Prometheus dcp stats emit both a bucket total metric in the low cardinality set, in addition to a per- connection_type metric in the high cardinality set - e.g.
$ http --auth Administrator:asdasd http://localhost:11996/_prometheusMetrics/ | rg items_sent
|
kv_dcp_items_sent{bucket="default",connection_type="replication"} 0.000000 1685006545000
|
$ http --auth Administrator:asdasd http://localhost:11996/_prometheusMetricsHigh/ | rg items_sent
|
kv_dcp_items_sent{bucket="default"} 0.000000 1685006536000
|
This is problematic for the few reasons:
- It results in graphs with an extra unlabelled series on them - e.g.
- It breaks calculating the sum of all connection_types - one gets an extra series for the total, which the user needs to be aware of and filter out.
(It also wastes a bit of space in Prometheus, but that's a pretty minor issue in the grand scheme of how many stats we track).
Full set of duplicated stats:
‡ for group in '' High; do http --auth Administrator:asdasd http://localhost:11996/_prometheusMetrics${group}; done | rg -o '^kv_dcp\w+'|sort| uniq -c| rg -v ' 1'
|
2 kv_dcp_count
|
2 kv_dcp_items_remaining
|
2 kv_dcp_items_sent
|
2 kv_dcp_total_data_size_bytes
|
2 kv_dcp_total_uncompressed_data_size_bytes
|
We should remove the duplicated metrics.