Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.0.0
-
Untriaged
-
1
-
Unknown
Description
On test installation in aws prometheus consumed ~25G of memory and got OOM killed:
level=info ts=2021-07-15T10:18:46.826Z caller=compact.go:494 component=tsdb msg="write block" mint=1626307200000 maxt=1626328800000 ulid=01FAMTT0VNVMV2149RSNW0M4GW duration=821.47751ms
|
level=info ts=2021-07-15T10:18:47.407Z caller=compact.go:494 component=tsdb msg="write block" mint=1626328800000 maxt=1626336000000 ulid=01FAMTT1NAT3G7FPGE729R1PBR duration=580.965225ms
|
level=info ts=2021-07-15T10:18:47.416Z caller=db.go:1152 component=tsdb msg="Deleting obsolete block" block=01FAMTR51NKY7WPP56W2201MMV
|
level=info ts=2021-07-15T10:18:47.420Z caller=db.go:1152 component=tsdb msg="Deleting obsolete block" block=01FAMTR5VBSYVN4G5XEWFRXZPZ
|
level=info ts=2021-07-15T10:18:47.436Z caller=db.go:1152 component=tsdb msg="Deleting obsolete block" block=01FAMTR3GW7WWJGKNF37FDXK60
|
fatal error: runtime: out of memory
|
runtime stack:
|
runtime.throw(0x28d6767, 0x16)
|
/home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/panic.go:1116 +0x72
|
runtime.sysMap(0xc6cc000000, 0x4000000, 0x45aa2d8)
|
/home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/mem_linux.go:169 +0xc5
|
runtime.(*mheap).sysAlloc(0x45954a0, 0x400000, 0x45954a8, 0xb9)
|
/home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/malloc.go:715 +0x1cd
|
runtime.(*mheap).grow(0x45954a0, 0xb9, 0x0)
|
/home/couchbase/jenkins/workspace/cbdeps-platform-build/deps/go1.14.2/src/runtime/mheap.go:1286 +0x11c
|
runtime.(*mheap).allocSpan(0x45954a0, 0xb9, 0xfc10100, 0x45aa2e8, 0xc004f6bf28)
|
Attachments
Issue Links
- Clones
-
MB-47502 Disable stats decimation as a 7.0.1 workaround for a memory leak in Prometheus
- Closed
Gerrit Reviews
For Gerrit Dashboard: MB-47816 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
159648,1 | MB-47816: Increment prometheus package vsn in order... | master | tlm | Status: ABANDONED | 0 | 0 |
159650,1 | MB-47816: Increment prometheus package vsn in order... | cheshire-cat | tlm | Status: ABANDONED | +2 | -1 |
159731,2 | MB-47816: Make cleaning stats tombstones configurable | cheshire-cat | ns_server | Status: MERGED | +2 | +1 |
159732,1 | MB-47816: Adjust default stats pruning settings | cheshire-cat | ns_server | Status: ABANDONED | +2 | -1 |
159824,1 | Merge remote-tracking branch 'couchbase/cheshire-cat' | master | ns_server | Status: MERGED | +2 | +1 |