Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37811

[CX] Analytics cluster down on IP reassignment

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 185

    Description

      As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.

      We should set networkaddress.cache.ttl to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes

       

      The above suggestion is insufficient, due to caching above the name service which is also a factor.

      Attachments

        Issue Links

          is parent task of

          Task - A task that needs to be done. MB-37834 [CX] Add IP address update tests (see MB-37811)

          • Major - Major loss of function.
          • Closed
          links to

          Web Link AsterixDB Gerrit Review

          Web Link AsterixDB Gerrit Review

          For Gerrit Dashboard: MB-37811
          # Subject Branch Project Status CR V
          121906,8 MB-37811: remove restart node workarounds mad-hatter cbas-core Status: MERGED +2 +1
          121907,3 MB-37811: configure InetAddress cache ttl to 15s mad-hatter cbas Status: MERGED +2 +1

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty