Prometheus continues to consume an ever increasing amount of disk space after 1GB

In MB-45843 a simple test is described that was used to investigate how disk usage due to stats collection grew over time.

To summarise the test involved:

Experiment 1

Created a single node cluster (cluster_run) on CC

With the following services

data

query

index

search

analytics

eventing

backup

and created 30 buckets, each with a 124 MiB quota.

Then left alone for ~16 days.

During this time we measure the amount of disk space in the stats directory using the following from the {{master/ns_server/data/n_0}} directory

{code}

while :; do date +%D:%H:%M:%S >> output.txt; du -sk stats_data >>output.txt ; sleep 300; done

{code}

The test led to two key observations:

- The disk usage increased at a rate much greater than expected.

- Due to the limit in the amount of disk space that can be used for stats the time period of which the stats covered was much smaller than expected.

For example, although the test ran for over 2 weeks, we only had data for approximately the last 48 hours.

Therefore we need to make improvements in this area - and has led to the creation of the EPIC - MB-47091.

To validate this work we need a longevity test to monitor the rate of disk space due to stats, and a measure of the time-period covered by the stats.

The experiment described above hopefully provides a good starting point. It is not expected that the test need to run for 16 days - instead a week should suffice.