Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47502

Disable stats decimation as a 7.0.1 workaround for a memory leak in Prometheus

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      On test installation in aws prometheus consumed ~25G of memory and got OOM killed:

      level=info ts=2021-07-15T10:18:46.826Z caller=compact.go:494 component=tsdb msg="write block" mint=1626307200000 maxt=1626328800000 ulid=01FAMTT0VNVMV2149RSNW0M4GW duration=821.47751ms
      level=info ts=2021-07-15T10:18:47.407Z caller=compact.go:494 component=tsdb msg="write block" mint=1626328800000 maxt=1626336000000 ulid=01FAMTT1NAT3G7FPGE729R1PBR duration=580.965225ms
      level=info ts=2021-07-15T10:18:47.416Z caller=db.go:1152 component=tsdb msg="Deleting obsolete block" block=01FAMTR51NKY7WPP56W2201MMV
      level=info ts=2021-07-15T10:18:47.420Z caller=db.go:1152 component=tsdb msg="Deleting obsolete block" block=01FAMTR5VBSYVN4G5XEWFRXZPZ
      level=info ts=2021-07-15T10:18:47.436Z caller=db.go:1152 component=tsdb msg="Deleting obsolete block" block=01FAMTR3GW7WWJGKNF37FDXK60
      fatal error: runtime: out of memory
      runtime stack:
      runtime.throw(0x28d6767, 0x16)
          /home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/panic.go:1116 +0x72
      runtime.sysMap(0xc6cc000000, 0x4000000, 0x45aa2d8)
          /home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/mem_linux.go:169 +0xc5
      runtime.(*mheap).sysAlloc(0x45954a0, 0x400000, 0x45954a8, 0xb9)
          /home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/malloc.go:715 +0x1cd
      runtime.(*mheap).grow(0x45954a0, 0xb9, 0x0)
          /home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/mheap.go:1286 +0x11c
      runtime.(*mheap).allocSpan(0x45954a0, 0xb9, 0xfc10100, 0x45aa2e8, 0xc004f6bf28)
      

      Logs: https://s3.amazonaws.com/cb-engineering/stevewatanabe-19JUL21-AWS/collectinfo-2021-07-19T165101-ns_1%40127.0.0.1.zip

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-47502
          # Subject Branch Project Status CR V

          Activity

            timofey.barmin Timofey Barmin created issue -
            timofey.barmin Timofey Barmin made changes -
            Field Original Value New Value
            Link This issue relates to MB-44878 [ MB-44878 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Assignee Meni Hillel [ JIRAUSER25407 ] Steve Watanabe [ steve.watanabe ]

            Assigning to Timofey as he's looking into this issue.

            steve.watanabe Steve Watanabe added a comment - Assigning to Timofey as he's looking into this issue.
            steve.watanabe Steve Watanabe made changes -
            Assignee Steve Watanabe [ steve.watanabe ] Timofey Barmin [ timofey.barmin ]
            meni.hillel Meni Hillel (Inactive) added a comment - From Prometheus : https://github.com/prometheus/prometheus/issues/9108#issuecomment-887894369

            This is the current workaround (Restart is not needed):
            curl 'http://Administrator:password@localhost:8091/settings/metrics/' -d 'truncation.enabled=false&decimation.enabled=false'

            meni.hillel Meni Hillel (Inactive) added a comment - This is the current workaround (Restart is not needed): curl 'http://Administrator:password@localhost:8091/settings/metrics/' -d 'truncation.enabled=false&decimation.enabled=false'
            James Flather James Flather added a comment - - edited

            Meni Hillel Am I correct in saying that the workaround will instead lead to disk growth instead (based on the option names)?

            James Flather James Flather added a comment - - edited Meni Hillel Am I correct in saying that the workaround will instead lead to disk growth instead (based on the option names)?

            James Flather Prometheus is configured with a limit to the amount of storage it will use. See "–storage.tsdb.retention.size" in https://prometheus.io/docs/prometheus/latest/storage/. When decimation is disabled there's no samples being deleted and so the storage will be used up sooner. As a result the length of time that stats are available will shorten.  The exact time is dependent on factors such as configuration and load.

            steve.watanabe Steve Watanabe added a comment - James Flather  Prometheus is configured with a limit to the amount of storage it will use. See "–storage.tsdb.retention.size" in https://prometheus.io/docs/prometheus/latest/storage/.  When decimation is disabled there's no samples being deleted and so the storage will be used up sooner. As a result the length of time that stats are available will shorten.  The exact time is dependent on factors such as configuration and load.

            James Flather No, disk consumption will stay the same. This workaround will lead to shorter stats history.

            timofey.barmin Timofey Barmin added a comment - James Flather No, disk consumption will stay the same. This workaround will lead to shorter stats history.
            meni.hillel Meni Hillel (Inactive) made changes -
            Link This issue blocks MB-47469 [ MB-47469 ]
            timofey.barmin Timofey Barmin made changes -
            Link This issue is cloned by MB-47816 [ MB-47816 ]
            timofey.barmin Timofey Barmin made changes -
            Summary Prometheus consumes inappropriate amount of memory and gets OOM killed Disable stats decimation as a workaround for 7.0.1 because of memory leak in Prometheus

            It seems the issue is being worked on as part of https://github.com/prometheus/prometheus/pull/9151.

            meni.hillel Meni Hillel (Inactive) added a comment - It seems the issue is being worked on as part of https://github.com/prometheus/prometheus/pull/9151 .
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            Fix Version/s 7.0.1 [ 17104 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Fix Version/s 7.0.2 [ 18012 ]

            Build couchbase-server-7.0.0-5304 contains ns_server commit 29e1058 with commit message:
            MB-47502: Temporarily disable stats pruning as it causes ...

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5304 contains ns_server commit 29e1058 with commit message: MB-47502 : Temporarily disable stats pruning as it causes ...
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.1 [ 17104 ]

            Build couchbase-server-7.0.0-5305 contains ns_server commit 4ab0277 with commit message:
            Revert "MB-47502: Temporarily disable stats pruning as it causes ..."

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5305 contains ns_server commit 4ab0277 with commit message: Revert " MB-47502 : Temporarily disable stats pruning as it causes ..."
            timofey.barmin Timofey Barmin made changes -
            Summary Disable stats decimation as a workaround for 7.0.1 because of memory leak in Prometheus Disable stats decimation as a 7.0.1 workaround for a memory leak in Prometheus

            Build couchbase-server-7.0.1-6004 contains ns_server commit 005cf4a with commit message:
            MB-47502: Temporarily disable stats pruning as it causes ...

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-6004 contains ns_server commit 005cf4a with commit message: MB-47502 : Temporarily disable stats pruning as it causes ...

            Build couchbase-server-7.0.1-6004 contains ns_server commit 4ab0277 with commit message:
            Revert "MB-47502: Temporarily disable stats pruning as it causes ..."

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-6004 contains ns_server commit 4ab0277 with commit message: Revert " MB-47502 : Temporarily disable stats pruning as it causes ..."

            Build couchbase-server-7.0.1-6004 contains ns_server commit 29e1058 with commit message:
            MB-47502: Temporarily disable stats pruning as it causes ...

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.1-6004 contains ns_server commit 29e1058 with commit message: MB-47502 : Temporarily disable stats pruning as it causes ...

            Build couchbase-server-7.1.0-1132 contains ns_server commit 005cf4a with commit message:
            MB-47502: Temporarily disable stats pruning as it causes ...

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1132 contains ns_server commit 005cf4a with commit message: MB-47502 : Temporarily disable stats pruning as it causes ...

            Build couchbase-server-7.1.0-1132 contains ns_server commit 4ab0277 with commit message:
            Revert "MB-47502: Temporarily disable stats pruning as it causes ..."

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1132 contains ns_server commit 4ab0277 with commit message: Revert " MB-47502 : Temporarily disable stats pruning as it causes ..."

            Build couchbase-server-7.1.0-1132 contains ns_server commit 29e1058 with commit message:
            MB-47502: Temporarily disable stats pruning as it causes ...

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1132 contains ns_server commit 29e1058 with commit message: MB-47502 : Temporarily disable stats pruning as it causes ...
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            meni.hillel Meni Hillel (Inactive) made changes -
            Fix Version/s 7.0.1 [ 17104 ]
            Fix Version/s 7.0.2 [ 18012 ]
            timofey.barmin Timofey Barmin made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Timofey Barmin - Is there any specific test that needs to be added to validate this defect. We plan to run the longevity + volume test + regular regression for validation. If there is any additional variation that needs to be done, then please let us know.

            ritam.sharma Ritam Sharma added a comment - Timofey Barmin - Is there any specific test that needs to be added to validate this defect. We plan to run the longevity + volume test + regular regression for validation. If there is any additional variation that needs to be done, then please let us know.

            Ritam Sharma The test is the following (we need to make sure prometheus is not leaking memory):

            • start a single node cluster with 30 buckets
            • make sure stats are collected and let it run for 4 days
            • check the sysproc_mem_resident {proc="prometheus"}

              metric and make sure the prometheus memory is not leaking (the graph should be flat and it should consume around 1-2GB or so).

            Note: you can use promtimer to connect to a working cluster and monitor metrics

            timofey.barmin Timofey Barmin added a comment - Ritam Sharma The test is the following (we need to make sure prometheus is not leaking memory): start a single node cluster with 30 buckets make sure stats are collected and let it run for 4 days check the sysproc_mem_resident {proc="prometheus"} metric and make sure the prometheus memory is not leaking (the graph should be flat and it should consume around 1-2GB or so). Note: you can use promtimer to connect to a working cluster and monitor metrics

            Setup :- 172.23.121.10:8091/

            Have been running the above setup for past 2 days and monitoring the following api

            http://172.23.121.10:8091/_prometheusMetrics + sysproc_mem_size{proc="prometheus",category="system-processes"}
            

            This number has been hovering around 1.5 GB for past 2 days. Will monitor this for few more days and mark it closed if it remains at this number based on the above comment.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Setup :- 172.23.121.10:8091/ Have been running the above setup for past 2 days and monitoring the following api http://172.23.121.10:8091/_prometheusMetrics + sysproc_mem_size{proc="prometheus",category="system-processes"} This number has been hovering around 1.5 GB for past 2 days. Will monitor this for few more days and mark it closed if it remains at this number based on the above comment.
            ritam.sharma Ritam Sharma made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 23033 ]
            Balakumaran.Gopal Balakumaran Gopal made changes -
            timofey.barmin Timofey Barmin added a comment - - edited

            Balakumaran Gopal Please wait several days more if possible. My explanation is the following: the more blocks it creates on disk, the more memory it uses. At some point it should start removing old blocks and memory should plateau. Hypothetically it should happen when (disk_size(./stats_data) - disk_size(./stats_data/WAL)) > 1GB.

            timofey.barmin Timofey Barmin added a comment - - edited Balakumaran Gopal Please wait several days more if possible. My explanation is the following: the more blocks it creates on disk, the more memory it uses. At some point it should start removing old blocks and memory should plateau. Hypothetically it should happen when (disk_size(./stats_data) - disk_size(./stats_data/WAL)) > 1GB.
            Balakumaran.Gopal Balakumaran Gopal made changes -

            Considering the original reported issue, where memory grows to very high levels 25G) after shorter duration, disable pruning seemed to have contributed to a significant improvement. Considering customers will be upgrading from 6.6 and/or initial evaluation of 7.x would get a much better resource consumption experience relative to 6.6, where we had very high memory consumption due to stats.

            This is not to say that we are satisfied with the current state. We'll continue to dig in and eliminate Prometheus continual memory growth to ensure it is capped. At this point the issue is clearly identified and acknowledged by Prometheus engineering. They have been working on the issue and provided a few patches, but have not merged them. We have experimented with these patches and determined they are still insufficient.

            Given above status, and per maintenance meeting, we recommend not to introduce any delay for releasing 7.0.1 and release as scheduled (1st week of Sep).

            meni.hillel Meni Hillel (Inactive) added a comment - Considering the original reported issue, where memory grows to very high levels 25G) after shorter duration, disable pruning seemed to have contributed to a significant improvement. Considering customers will be upgrading from 6.6 and/or initial evaluation of 7.x would get a much better resource consumption experience relative to 6.6, where we had very high memory consumption due to stats. This is not to say that we are satisfied with the current state. We'll continue to dig in and eliminate Prometheus continual memory growth to ensure it is capped. At this point the issue is clearly identified and acknowledged by Prometheus engineering. They have been working on the issue and provided a few patches, but have not merged them. We have experimented with these patches and determined they are still insufficient. Given above status, and per maintenance meeting, we recommend not to introduce any delay for releasing 7.0.1 and release as scheduled (1st week of Sep).
            wayne Wayne Siu made changes -
            Labels approved-for-7.0.1
            wayne Wayne Siu made changes -
            Assignee Timofey Barmin [ timofey.barmin ] Ritam Sharma [ ritam.sharma ]

            There is a question about virtual memory consumption on github: https://github.com/prometheus/prometheus/issues/5295
            I think we should look at resident memory instead. It seems like retention policy started working today, so it should plateau soon I hope.

            timofey.barmin Timofey Barmin added a comment - There is a question about virtual memory consumption on github: https://github.com/prometheus/prometheus/issues/5295 I think we should look at resident memory instead. It seems like retention policy started working today, so it should plateau soon I hope.

            Actually I was wrong. Retention.size has not started working there yet. The total stats_data size is 1.3GB and blocks are ~100MB in size, so it should have removed 3 blocks already, but that's not happening. On another server we see similar behavior with 2GB total size. Here is a bug about that and it seems like it is expected behavior. I remember we saw it working before though. We can only wait I think, it seems like it should start removing old blocks.

            timofey.barmin Timofey Barmin added a comment - Actually I was wrong. Retention.size has not started working there yet. The total stats_data size is 1.3GB and blocks are ~100MB in size, so it should have removed 3 blocks already, but that's not happening. On another server we see similar behavior with 2GB total size. Here is a bug about that and it seems like it is expected behavior. I remember we saw it working before though. We can only wait I think, it seems like it should start removing old blocks.

            We've reviewed current state of resident memory and we are satisfied with it. Virtual memory still grows as it is mapped to files and it should be ok. The resident memory is the one we are most concern as high consumption will end up with OOM on Prometheus or cause other OS to refuse memory allocation for other processes. We are good to go for 7.0.1.

            meni.hillel Meni Hillel (Inactive) added a comment - We've reviewed current state of resident memory and we are satisfied with it. Virtual memory still grows as it is mapped to files and it should be ok. The resident memory is the one we are most concern as high consumption will end up with OOM on Prometheus or cause other OS to refuse memory allocation for other processes. We are good to go for 7.0.1.

            Closing this based on above comments.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Closing this based on above comments.
            Balakumaran.Gopal Balakumaran Gopal made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            malarky Chris Malarky made changes -
            Link This issue relates to CBSP-3994 [ CBSP-3994 ]
            drigby Dave Rigby made changes -
            Link This issue causes CBSE-10772 [ CBSE-10772 ]

            People

              ritam.sharma Ritam Sharma
              timofey.barmin Timofey Barmin
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty