Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.0
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.1.0-2021
Test : -test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
Scale : 3
Iteration : 1st
On 172.23.104.157, seeing an OOM kill at Nov 12 15:01:38.
From dmesg logs on the VM :
[Mon Jan 10 19:09:49 2022] Out of memory: Kill process 4756 (cbq-engine) score 895 or sacrifice child
|
[Mon Jan 10 19:09:49 2022] Killed process 4756 (cbq-engine) total-vm:70423096kB, anon-rss:22666908kB, file-rss:0kB, shmem-rss:0kB
|
Seeing these warnings in the log a few seconds before the crash :
2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
|
2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
|
2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
|
2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `041d02c8bf6c2ca9/dcaf4ae236a29b3f` : EOF
|
2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `d1e7bc6c83165164/7f0babb77d0ecd65` : EOF
|
2022-01-10T19:09:23.075-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `d1e7bc6c83165164/e24fb1651c74a924` : EOF
|
2022-01-10T19:09:23.075-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `d1104ad1ba63e3ad/80735d5e741a70d8` : EOF
|
2022-01-10T19:09:23.075-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
|
2022-01-10T19:09:23.082-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc195e9cd20 failed to bootstrap: bucket not found
|
2022-01-10T19:09:23.140-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc14fbdc960 failed to bootstrap: bucket not found
|
2022-01-10T19:09:23.144-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc1324c6a80 failed to bootstrap: bucket not found
|
2022-01-10T19:09:23.148-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc6ee4862a0 failed to bootstrap: bucket not found
|
2022-01-10T19:09:23.148-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc007dfd620 failed to bootstrap: bucket not found
|
Query nodes : 172.23.104.155, 172.23.104.157
Is this similar to MB-49553?
Heap dumps and goroutine dumps from 172.23.104.157 are collected just before the crash and saved at /root/heap_gr.tar.gz on 172.23.104.157 (cannot attach due to file size)
Attachments
Issue Links
- is caused by
-
GOCBC-1222 Use a sync.pool for memd.conn readers
- Resolved
- relates to
-
GOCBC-1238 Allow connection reader buffer size to be set
- Resolved