Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.6.2
-
Couchbase Server Enterprise Edition 7.6.2 build 3573
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
Description
In Couchbase server version 7.6.2 and after, xattrs mappings are introduced. But after indexing 'xattrs' fields, the prometheus fails to scrape the metrics from xattrs fields.
Steps to reproduce the error
- Create a 6 node cluster. (cluster_services ::['kv', 'kv', 'kv', 'fts', 'fts', 'fts']).
- Create a bucket 'standard_bkt'. Load 1000000 (1M) docs into the bucket.
- The schema of the document is as follows:
Data : { id : documentID, sno : unique_no, sname : random_name }
Metadata : [ *xattrs : \{ "vector_data" : <vector_embedding_1536_dimensions> }* ]
- Create a search index "xattrs_search_index", indexing the field 'vector_data' in xattrs section in metadata.
- Download the cbcollect_info.zip from any one of the search node.
- In cbcollect_info locate the file ns_server.prometheus.log.
ns_server.prometheus.log ( a chunk of ns_server.prometheus.log to describle the problem ) :
ts=2024-04-24T09:04:24.453Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:14:14.098Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:17:08.800Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=general target="http://127.0.0.1:8094/_prometheusMetrics?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetrics?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:18:23.629Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:20:43.492Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:22:17.211Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:22:17.211Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=general target="http://127.0.0.1:8094/_prometheusMetrics?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetrics?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:25:28.858Z caller=scrape.go:1372 level=debug component="scrape manager" scrape_pool=general target="http://127.0.0.1:8094/_prometheusMetrics?timeout=10000" msg="Scrape failed" err="Get \"http://127.0.0.1:8094/_prometheusMetrics?timeout=10000\": dial tcp 127.0.0.1:8094: connect: connection refused"ts=2024-04-24T09:49:13.343Z caller=scrape.go:1387 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Append failed" err="invalid metric type \"xattrs.vector_data:num_vectors gauge\""ts=2024-04-24T09:49:23.340Z caller=scrape.go:1387 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Append failed" err="invalid metric type \"xattrs.vector_data:num_vectors gauge\""ts=2024-04-24T09:49:33.236Z caller=scrape.go:1387 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Append failed" err="invalid metric type \"xattrs.vector_data:num_vectors gauge\"" |
and the error occurs here,
ts=2024-04-24T09:49:23.340Z caller=scrape.go:1387 level=debug component="scrape manager" scrape_pool=fts_high_cardinality target="http://127.0.0.1:8094/_prometheusMetricsHigh?timeout=10000" msg="Append failed" err="invalid metric type \"xattrs.vector_data:num_vectors gauge\"" |
even though we are now able to do vector search with the vector data being stored in xattrs, the prometheus identifies it as invalid metric type ( assuming that it didn't get updated with the new features of 7.6.2 )