Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55100

Global cluster config not sent to memcached after memcached is OOM-killed and restarted

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      SDK tries to get configuration via KV connection, before selecting bucket, but the server returns NOT_FOUND (see tcp dump in the attachment).

      This behaviour is reproducible with cluster "10.1.28.117:31995" only. Another cluster that uses same version works. Also this particular cluster reported as operational several days ago.

      From looking at the logs, memcahced was restarted on 10th January due to the Linux OOM-killer terminating it. After restart, no global config is sent to memcached (per-bucket config is correctly sent):

      $ rg -N '(Couchbase version|Updated global cluster configuration)' memcached.log 
      2023-01-10T14:39:51.635909+00:00 INFO Couchbase version 7.1.2-3454 starting.
      

      Note the lack of "Updated global cluster configuration" message, which is seen during initial CB Server startup (i.e. when babysitter and ns_server starts) - example from a different 7.1.1 system:

      # grep -E '(Couchbase version|Updated global cluster configuration)' memcached.log.000084.txt 
      2023-01-13T17:12:33.387179+00:00 INFO Couchbase version 7.1.1-0000 starting.
      2023-01-13T17:12:34.112813+00:00 INFO 62: [ {"ip":"127.0.0.1","port":47476} - {"ip":"127.0.0.1","port":11209} (System, <ud>@ns_server</ud>) ] Updated global cluster configuration. New revision: {"epoch":1,"revno":4095}
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-55100
          # Subject Branch Project Status CR V

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              avsej Sergey Avseyev
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty