Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42360

Displaying doc items count and TTL for collections in the UI

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • UI
    • 1

    Description

      As discussed in the UI weekly meeting, filing this improvement to display doc items count and TTL for collections in the UI - this is a P1 request

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            rob.ashcom Rob Ashcom added a comment -

            Timofey Barmin  is this information accessible to the UI? 

            rob.ashcom Rob Ashcom added a comment - Timofey Barmin   is this information accessible to the UI? 
            timofey.barmin Timofey Barmin added a comment - - edited

            Rob Ashcom

            There is a kv_items metric reported by kv. I think you can use it to retrieve items count. Example:

            $ curl -X GET -u Administrator:password 'http://127.0.0.1:9000/pools/default/stats/range/kv_items?bucket=test&scope=my_scope&collection=my_collection'
            
            

             

            The problem is that right now kv uses collection and scope id's (instead of names) when reporting this metric. I think it is not correct and I plan to create a ticket to fix that.

            Unfortunately I don't know anything about TTL.

            Daniel Owen James Harrison Do you report ttl for collections to prometheus?

             

            timofey.barmin Timofey Barmin added a comment - - edited Rob Ashcom There is a kv_items metric reported by kv. I think you can use it to retrieve items count. Example: $ curl -X GET -u Administrator:password 'http://127.0.0.1:9000/pools/default/stats/range/kv_items?bucket=test&scope=my_scope&collection=my_collection'   The problem is that right now kv uses collection and scope id's (instead of names) when reporting this metric. I think it is not correct and I plan to create a ticket to fix that. Unfortunately I don't know anything about TTL. Daniel Owen James Harrison  Do you report ttl for collections to prometheus?  
            james.harrison James Harrison added a comment - - edited

            Collection maxTTL is reported by KV to Prometheus as

            maxTTL{bucket="...", scope="...", collection="..."}
            

            This will only be exposed from KV once a collection with a maxTTL has been created.

            However, ns_server is the source of truth for the maxTTL; it is provided to KV as part of the manifest. To avoid a loop through KV and Prometheus (and any potential failure cases that adds) it can be fetched directly from ns_server

            For a bucket named "FOOBAR" :

            $ curl -s -u Administrator:asdasd http://localhost:9000/pools/default/buckets/FOOBAR/collections
            {"uid":"0","scopes":[{"name":"_default","uid":"0","collections":[{"name":"_default","uid":"0","maxTTL":0}]}]}
            

            jq pretty printed

            {
              "uid": "0",
              "scopes": [
                {
                  "name": "_default",
                  "uid": "0",
                  "collections": [
                    {
                      "name": "_default",
                      "uid": "0",
                      "maxTTL": 0
                    }
                  ]
                }
              ]
            }
            

            The per-collection maxTTL is present there. There may be other/more manifest related endpoints, but the ns_server team will have more info on that.

            james.harrison James Harrison added a comment - - edited Collection maxTTL is reported by KV to Prometheus as maxTTL{bucket="...", scope="...", collection="..."} This will only be exposed from KV once a collection with a maxTTL has been created. However, ns_server is the source of truth for the maxTTL; it is provided to KV as part of the manifest. To avoid a loop through KV and Prometheus (and any potential failure cases that adds) it can be fetched directly from ns_server For a bucket named "FOOBAR" : $ curl -s -u Administrator:asdasd http://localhost:9000/pools/default/buckets/FOOBAR/collections {"uid":"0","scopes":[{"name":"_default","uid":"0","collections":[{"name":"_default","uid":"0","maxTTL":0}]}]} jq pretty printed { "uid": "0", "scopes": [ { "name": "_default", "uid": "0", "collections": [ { "name": "_default", "uid": "0", "maxTTL": 0 } ] } ] } The per-collection maxTTL is present there. There may be other/more manifest related endpoints, but the ns_server team will have more info on that.
            dfinlay Dave Finlay added a comment -

            Thanks Timofey & James.

            Pavel Blagodov: you'll need to use the stats API that Timofey mentions and the collections REST API that James mentions to get both of these pieces of information. Note that the collections REST APIs are changing slightly as captured here: MB-42712. Make sure to use the new versions of the API.

            dfinlay Dave Finlay added a comment - Thanks Timofey & James. Pavel Blagodov : you'll need to use the stats API that Timofey mentions and the collections REST API that James mentions to get both of these pieces of information. Note that the collections REST APIs are changing slightly as captured here: MB-42712 . Make sure to use the new versions of the API.

            The UI (attached screenshot) also needs to show the following stats per scope and per collection (in addition to item count):

            • memory used
            • disk used
            • ops/sec

            ops/sec is comprised of 3 different stats - up to you whether you want to break those down or not.

            Jim Walker fyi

            shivani.gupta Shivani Gupta added a comment - The UI (attached screenshot) also needs to show the following stats per scope and per collection (in addition to item count): memory used disk used ops/sec ops/sec is comprised of 3 different stats - up to you whether you want to break those down or not. Jim Walker  fyi
            timofey.barmin Timofey Barmin added a comment - - edited

            Pavel Blagodov

            You can use the following metrics for that:
            kv_mem_used_bytes{bucket="travel-sample", collection="my_collection", scope="my_scope"}
            kv_disk_size_bytes{bucket="travel-sample",collection="my_collection",scope="my_scope"}
            kv_collection_ops{bucket="travel-sample",collection="my_collection",op="store",scope="my_scope"}
            kv_collection_ops{bucket="travel-sample",collection="my_collection",op="get",scope="my_scope"}
            kv_collection_ops{bucket="travel-sample",collection="my_collection",op="delete",scope="my_scope"}

            If you decide to show aggregated ops, you can calculate sum across 'op' label using api. Let me know if you you need help with that.

             

            UPDATE: Note that kv_mem_used_bytes might be renames as a result of MB-43101

            timofey.barmin Timofey Barmin added a comment - - edited Pavel Blagodov You can use the following metrics for that: kv_mem_used_bytes{bucket="travel-sample", collection="my_collection", scope="my_scope"} kv_disk_size_bytes{bucket="travel-sample",collection="my_collection",scope="my_scope"} kv_collection_ops{bucket="travel-sample",collection="my_collection",op="store",scope="my_scope"} kv_collection_ops{bucket="travel-sample",collection="my_collection",op="get",scope="my_scope"} kv_collection_ops{bucket="travel-sample",collection="my_collection",op="delete",scope="my_scope"} If you decide to show aggregated ops, you can calculate sum across 'op' label using api. Let me know if you you need help with that.   UPDATE: Note that kv_mem_used_bytes might be renames as a result of MB-43101

            Build couchbase-server-7.0.0-4322 contains ns_server commit 9a0a420 with commit message:
            MB-42360: add stats to scopes and collections page

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4322 contains ns_server commit 9a0a420 with commit message: MB-42360 : add stats to scopes and collections page

            Bug bashes conducted to test this feature

            arunkumar Arunkumar Senthilnathan added a comment - Bug bashes conducted to test this feature

            People

              pavel Pavel Blagodov
              arunkumar Arunkumar Senthilnathan
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty