Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48239

High-card stats are not getting deleted

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      High cardinality stats are not getting deleted after 3 days. In this code "LastPruningTime" gets updated every time the pruning pid runs. So LastPruningTime will be larger than Now and End will be negative.

      run_truncate_stats(LastPruningTime, Settings) ->
          Now = os:system_time(seconds),
          MaxAge = proplists:get_value(truncate_max_age, Settings),
          %% Amount of time to not truncate stats even if they're older than the
          %% maximum age.
          MinTruncationInterval = proplists:get_value(min_truncation_interval,
                                                      Settings),
          End = Now - MaxAge,
          %% Each call truncates the little bit that has exceeded the age limit
          %% since the last call.  We might want to do this less frequently e.g.
          %% when a certain time frame is exceeded.
          case End - LastPruningTime > MinTruncationInterval of
              true ->
                  do_truncate_stats(LastPruningTime, End, Settings),
                  true;
              false ->
                  false
          end.
      

      One possibility is to not specify "start" as the prometheus docs say...

      start=<rfc3339 | unix_timestamp>: Start timestamp. Optional and defaults to minimum possible time.
      

      But I believe I had a "start" ..probably an optimization. The minimal possible time would be the oldest stat that prometheus knows about...which could be a months old. So every time truncation occurs it would delete stats for the period of "months ago" to "3 days ago". Even though the months old stats would be low-card stats.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            steve.watanabe Steve Watanabe
            steve.watanabe Steve Watanabe
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty