Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37811

[CX] Analytics cluster down on IP reassignment

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Is this a Regression?:
      Unknown
    • Sprint:
      CX Sprint 185

      Description

      As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.

      We should set networkaddress.cache.ttl to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes

       

      The above suggestion is insufficient, due to caching above the name service which is also a factor.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            mihir.kamdar Mihir Kamdar added a comment -

            Arunkumar Senthilnathan can you pls take a look at this - it is K8S related.

            Show
            mihir.kamdar Mihir Kamdar added a comment - Arunkumar Senthilnathan can you pls take a look at this - it is K8S related.
            Hide
            arunkumar Arunkumar Senthilnathan added a comment -

            Covered by regression tests written by devs

            Show
            arunkumar Arunkumar Senthilnathan added a comment - Covered by regression tests written by devs
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-1006.5.1-1125 contains cbas commit c5f4aa7 with commit message:
            MB-37811: configure InetAddress cache ttl to 15s

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-1006.5.1-1125 contains cbas commit c5f4aa7 with commit message: MB-37811 : configure InetAddress cache ttl to 15s
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-1006.5.1-1125 contains cbas-core commit 9cb70ef with commit message:
            MB-37811: remove restart node workarounds

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-1006.5.1-1125 contains cbas-core commit 9cb70ef with commit message: MB-37811 : remove restart node workarounds
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit 9cb70ef with commit message:
            MB-37811: remove restart node workarounds

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 9cb70ef with commit message: MB-37811 : remove restart node workarounds

              People

              Assignee:
              arunkumar Arunkumar Senthilnathan
              Reporter:
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty