Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.0.0
-
Untriaged
-
1
-
Unknown
Description
High cardinality stats are not getting deleted after 3 days. In this code "LastPruningTime" gets updated every time the pruning pid runs. So LastPruningTime will be larger than Now and End will be negative.
run_truncate_stats(LastPruningTime, Settings) -> |
Now = os:system_time(seconds), |
MaxAge = proplists:get_value(truncate_max_age, Settings), |
%% Amount of time to not truncate stats even if they're older than the |
%% maximum age. |
MinTruncationInterval = proplists:get_value(min_truncation_interval, |
Settings), |
End = Now - MaxAge, |
%% Each call truncates the little bit that has exceeded the age limit |
%% since the last call. We might want to do this less frequently e.g. |
%% when a certain time frame is exceeded. |
case End - LastPruningTime > MinTruncationInterval of |
true ->
|
do_truncate_stats(LastPruningTime, End, Settings), |
true;
|
false ->
|
false
|
end. |
One possibility is to not specify "start" as the prometheus docs say...
start=<rfc3339 | unix_timestamp>: Start timestamp. Optional and defaults to minimum possible time.
|
But I believe I had a "start" ..probably an optimization. The minimal possible time would be the oldest stat that prometheus knows about...which could be a months old. So every time truncation occurs it would delete stats for the period of "months ago" to "3 days ago". Even though the months old stats would be low-card stats.