Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48728

[System Test] XDCR OOM killed multiple times

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1395
      Test : -test tests/integration/neo/test_neo_magma_wo_gsi_n1ql.yml -scope tests/integration/neo/scope_neo_magma_wo_gsi_n1ql.yml
      Scale : 3
      Iteration : 1st

      This is a new longevity test with all buckets having Magma storage. The cluster has only KV nodes and 1 backup node.

      It has been observed that the XDCR process has been OOM killed on multiple nodes at different times.

      Following are the excerpts from the eagle-eye tool :

      172.23.120.77 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T08:03:03.136-07:00,ns_1@172.23.120.77:<0.14628.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.97.241 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T08:26:04.623-07:00,ns_1@172.23.97.241:<0.23362.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.123.24 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T09:53:19.298-07:00,ns_1@172.23.123.24:<0.16777.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.123.25 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T09:58:25.640-07:00,ns_1@172.23.123.25:<0.15535.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.97.74 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T11:04:06.837-07:00,ns_1@172.23.97.74:<0.25407.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.120.86 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T13:38:47.900-07:00,ns_1@172.23.120.86:<0.14015.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.121.77 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T12:53:33.652-07:00,ns_1@172.23.121.77:<0.19069.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.96.122 : crash
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-10-02T13:28:12.842-07:00,ns_1@172.23.96.122:<0.25859.0>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
       
      172.23.96.14 : crash
      [user:info,2021-10-02T16:52:07.759-07:00,ns_1@172.23.96.14:<0.28392.20>:ns_log:crash_consumption_loop:72]Service 'goxdcr' exited with status 137. Restarting. Messages:
      

      Attached logs are the latest so far. If you need logs from an earlier timestamp, please let me know.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty