Major Description
ns_server keeps these stats in an ets table and reports them when prometheus does a scrape. ns_server will continue to report these stats even if the bucket is deleted. If the node is rebooted, the ets table is reset and ns_server will no longer report the stats to prometheus when it does a scrape. However prometheus still knows about these stats and will report them until it declares them as stale (10 minutes configured via --query.lookback-delta). Here's a picture of a reproduction I did on my macbook followed by an explanation.
Two buckets are present: one is retained and one is deleted. The "DELETION" of the "deleted bucket" occurs at 13:31. However the stats associated with the bucket remain in the ets table and continue to be reported to prometheus via its scrapes. The node was rebooted ~13:38:30 which resulted in the ets table getting reset. At this point ns_server no longer reports the stats to prometheus scrapes. However prometheus has recorded the stats and will continue to report them until the --query.lookback-delta (10 minutes) time expires. This is "Stale stat no longer displayed".
The stats associated with a deleted bucket should be removed from the ets table so they are no longer scraped by promethes.
Attachments
| For Gerrit Dashboard: MB-60149 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 202702,13 | MB-60149 Remove stats when bucket is deleted | master | ns_server | Status: MERGED | +2 | +1 |