Details
-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
None
Description
By default the metrics that are ingested for indexes look similar to the following:
index_num_requests{bucket="flightdata", cluster_name="Demo Cluster 3", collection="_default", index="adv_IATA_docType_active_ICAO_name_callsign", instance="host1", job="couchbase", scope="_default"} |
index_num_requests{bucket="flightdata", cluster_name="Demo Cluster 3", collection="_default", index="adv_IATA_docType_active_ICAO_name_callsign (replica 1)", instance="host2", job="couchbase", scope="_default"} |
This makes detecting indexes replicas for example, more convoluted but still it can be done. Using the existing metrics as an example you can find the # of replicas with a PromQL statement like so
count without (instance) (label_replace(index_items_count, "index", "$1", "index", "([^ ]+).*")) - 1 |
Ideally if you're filtering down to an index, you would want to see all versions of that index both active and replicas in the same graph, which means you would have to use a label_replace() like above which just seems backwards.
The :8091/indexStatus endpoint as an example has attributes for index, and indexName using the above indexes this would yield
index: adv_IATA_docType_active_ICAO_name_callsign
|
indexName: adv_IATA_docType_active_ICAO_name_callsign
|
|
|
index: adv_IATA_docType_active_ICAO_name_callsign (replica 1) |
indexName: adv_IATA_docType_active_ICAO_name_callsign
|
We can achieve this same behavior, as well as add a replicaId attribute which is not on the :8091/indexStatus endpoint but is present in the indexer admin stats. To do this we'd add a metric_relabel_config to the couchbase job which keeps the existing functionality the same but allows us to build on to it:
metric_relabel_configs:
|
# create indexName label that does not have the replica information if present |
- source_labels: [ index ]
|
action: replace
|
regex: "([^\\s]+).*" |
replacement: "$1" |
target_label: indexName
|
# set default replicaId to "0" |
- source_labels: [ __name__ ]
|
action: replace
|
regex: ".*" |
replacement: "0" |
target_label: replicaId
|
# override the replicaId if the name follows: idx_name (replica 1) |
- source_labels: [ index ]
|
action: replace
|
regex: "([^\\s]+)\\s+\\([A-Za-z]+\\s+([0-9]+)\\)" |
replacement: "$2" |
target_label: replicaId
|
Now the metrics would look like:
index_num_requests{bucket="flightdata", cluster_name="Demo Cluster 3", collection="_default", index="adv_IATA_docType_active_ICAO_name_callsign", indexName="adv_IATA_docType_active_ICAO_name_callsign", instance="host1", job="couchbase", replicaId="0", scope="_default"} |
index_num_requests{bucket="flightdata", cluster_name="Demo Cluster 3", collection="_default", index="adv_IATA_docType_active_ICAO_name_callsign (replica 1)", indexName="adv_IATA_docType_active_ICAO_name_callsign", instance="host2", job="couchbase", replicaId="1", scope="_default"} |
Now finding the number of replicas is a far simpler query
count(index_num_requests) by (bucket, indexName)
|
Or showing the panel with the number of requests for an index (both active and replica) is just another query
rate(index_num_requests{indexName="adv_IATA_docType_active_ICAO_name_callsign"}[5m]) |