Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50332

[System Test] cbq-engine OOM killed

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-2021
      Test : -test tests/integration/neo/test_neo_couchstore_milestone4.yml -scope tests/integration/neo/scope_couchstore.yml
      Scale : 3
      Iteration : 1st

      On 172.23.104.157, seeing an OOM kill at Nov 12 15:01:38.

      From dmesg logs on the VM :

      [Mon Jan 10 19:09:49 2022] Out of memory: Kill process 4756 (cbq-engine) score 895 or sacrifice child
      		 
      [Mon Jan 10 19:09:49 2022] Killed process 4756 (cbq-engine) total-vm:70423096kB, anon-rss:22666908kB, file-rss:0kB, shmem-rss:0kB

      Seeing these warnings in the log a few seconds before the crash :

      2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
      		 
      2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
      		 
      2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
      		 
      2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `041d02c8bf6c2ca9/dcaf4ae236a29b3f` : EOF
      		 
      2022-01-10T19:09:23.074-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `d1e7bc6c83165164/7f0babb77d0ecd65` : EOF
      		 
      2022-01-10T19:09:23.075-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `d1e7bc6c83165164/e24fb1651c74a924` : EOF
      		 
      2022-01-10T19:09:23.075-08:00 [WARN] (TXGOCBCORE) memdClient read failure on conn `d1104ad1ba63e3ad/80735d5e741a70d8` : EOF
      		 
      2022-01-10T19:09:23.075-08:00 [WARN] (TXGOCBCORE) Failed to connect to host. specific server is invalid
      		 
      2022-01-10T19:09:23.082-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc195e9cd20 failed to bootstrap: bucket not found
      		 
      2022-01-10T19:09:23.140-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc14fbdc960 failed to bootstrap: bucket not found
      		 
      2022-01-10T19:09:23.144-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc1324c6a80 failed to bootstrap: bucket not found
      		 
      2022-01-10T19:09:23.148-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc6ee4862a0 failed to bootstrap: bucket not found
      		 
      2022-01-10T19:09:23.148-08:00 [WARN] (TXGOCBCORE) Pipeline Client 0xc007dfd620 failed to bootstrap: bucket not found

      Query nodes : 172.23.104.155, 172.23.104.157

      Is this similar to MB-49553?

      Heap dumps and goroutine dumps from 172.23.104.157 are collected just before the crash and saved at /root/heap_gr.tar.gz on 172.23.104.157 (cannot attach due to file size)

      Attachments

        1. heapdumps.tar
          1.53 MB
        2. heapstackdumps_0201.zip
          3.85 MB
        3. MB50332repro.zip
          6.79 MB
        4. query.log
          354 kB

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. GOCBC-1222 Use a sync.pool for memd.conn readers

          • Major - Major loss of function.
          • Resolved
          relates to

          New Feature - A new feature of the product, which has yet to be developed. GOCBC-1238 Allow connection reader buffer size to be set

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty