Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.0.1
-
Untriaged
-
-
1
-
Yes
Description
Build : 7.0.1-5977
Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_recovery.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_dgm.yml
Scale : 2
Iteration : 3rd
On 172.23.97.236, seeing cbq-engine crash due to OOM
2021-08-03T16:40:04.547-07:00 [Info] GsiClient::UpdateUsecjson: using collatejson as data format between indexer and GsiClient
|
fatal error: runtime: out of memory
|
|
runtime stack:
|
runtime.throw(0x2512146, 0x16)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:774 +0x72
|
runtime.sysMap(0xc600000000, 0x4000000, 0x3b5f4d8)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mem_linux.go:169 +0xc5
|
runtime.(*mheap).sysAlloc(0x3b46220, 0x2000, 0x2000, 0x1d27fbda0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/malloc.go:701 +0x1cd
|
runtime.(*mheap).grow(0x3b46220, 0x1, 0xffffffff)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1255 +0xa3
|
runtime.(*mheap).allocSpanLocked(0x3b46220, 0x1, 0x3b5f4e8, 0xc000053320)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1170 +0x266
|
runtime.(*mheap).alloc_m(0x3b46220, 0x1, 0xb, 0xc5ffe9b200)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1022 +0xc2
|
runtime.(*mheap).alloc.func1()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/mheap.go:1093 +0x4c
|
runtime.systemstack(0x0)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/asm_amd64.s:370 +0x66
|
runtime.mstart()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/proc.go:1146
|
At this time, the test is dropping all the indexes as it is the last stage of the iteration.
If you look at the available memory chart for the node, it has been very low since ~5 AM on 8/3.
Attached are the logs after the crash happened, and also the various dumps and profiles collected at about 9 PM on 8/3, while the memory usage on the node was still high - 98%
The other query node is 172.23.97.227.
Let me know if you need logs from an earlier timestamp.