Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.1.0-1701
Test : -test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
Scale : 3
Iteration : 1
On 172.23.97.149, seeing an OOM kill at Nov 12 15:01:38.
From dmesg logs on the VM :
[Fri Nov 12 15:01:38 2021] Out of memory: Kill process 53020 (cbq-engine) score 917 or sacrifice child
|
[Fri Nov 12 15:01:38 2021] Killed process 53020 (cbq-engine), UID 996, total-vm:43103336kB, anon-rss:21097044kB, file-rss:0kB, shmem-rss:0kB
|
Couldnt find anything obvious in the query log. However, seeing these warnings in the log a couple of mins before the crash :
2021-11-12T14:59:53.459-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0x1c0bccd9f80 failed to bootstrap: bucket not found
|
2021-11-12T14:59:53.459-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `010889fa56a1bfa7/e8626c95013c146b` : EOF
|
2021-11-12T14:59:53.459-08:00 [INFO] (TXGOCBCORE) Stopping CCCP poller for HTTP polling takeover
|
2021-11-12T14:59:53.460-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `cf13dc92119eb372/7f3133cfd1ffddf2` : EOF
|
2021-11-12T14:59:53.460-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `4eb03ec5a6b9fedd/9b948606bf001402` : EOF
|
2021-11-12T14:59:53.460-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `80ebadc51ae669b0/ce387dde6f89ea8c` : EOF
|
2021-11-12T14:59:53.461-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `4eb03ec5a6b9fedd/d70328b1590e3e41` : EOF
|
2021-11-12T14:59:53.461-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `4eb03ec5a6b9fedd/b9e6216e5f686d6f` : EOF
|
The logs attached were collected after 2021-11-12 15:27. The only cbcollect done before this was at 2021-11-12 12:59 - not sure if that would be helpful in this case.
Query nodes : 172.23.97.150 & 172.23.97.149