Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37811

[CX] Analytics cluster down on IP reassignment

    XMLWordPrintable

Details

    • Untriaged
    • Unknown
    • CX Sprint 185

    Description

      As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.

      We should set networkaddress.cache.ttl to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes

       

      The above suggestion is insufficient, due to caching above the name service which is also a factor.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            michael.blow Michael Blow created issue -
            michael.blow Michael Blow made changes -
            Field Original Value New Value
            Link This issue causes CBSE-7909 [ CBSE-7909 ]
            michael.blow Michael Blow made changes -
            Labels 6.5.1-candidate 6.x-candidate 6.5.1-candidate 6.x-candidate kubernetes
            till Till Westmann made changes -
            Rank Ranked higher
            michael.blow Michael Blow made changes -
            Rank Ranked higher
            michael.blow Michael Blow made changes -
            Sprint CX Sprint 185 [ 983 ]
            michael.blow Michael Blow made changes -
            Rank Ranked lower
            till Till Westmann made changes -
            Labels 6.5.1-candidate 6.x-candidate kubernetes 6.5.1-candidate 6.x-candidate kubernetes releasenote
            michael.blow Michael Blow made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            michael.blow Michael Blow made changes -
            Labels 6.5.1-candidate 6.x-candidate kubernetes releasenote 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote
            till Till Westmann made changes -
            Labels 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote triaged
            michael.blow Michael Blow made changes -
            Labels 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote triaged 6.0.5-candidate 6.5.1-candidate 6.x-candidate DBaaS kubernetes releasenote triaged
            wayne Wayne Siu made changes -
            Affects Version/s Mad-Hatter [ 15037 ]
            Affects Version/s 6.5.0 [ 16624 ]
            michael.blow Michael Blow made changes -
            Summary [CX] Infinite DNS lookup address caching breaks clusters on IP reassignment [CX] Analytics cluster down on IP reassignment
            michael.blow Michael Blow made changes -
            Description As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.

            We should set {{networkaddress.cache.ttl}} to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes
            As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.

            -We should set {{networkaddress.cache.ttl}} to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes-

             

            The above suggestion is insufficient, due to caching above the name service which is also a factor.
            michael.blow Michael Blow made changes -
            Link This issue is parent task of MB-37834 [ MB-37834 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-37192 [ MB-37192 ]
            michael.blow Michael Blow made changes -
            Remote Link This issue links to "AsterixDB Gerrit Review (Web Link)" [ 19202 ]
            michael.blow Michael Blow made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            wayne Wayne Siu made changes -
            Labels 6.0.5-candidate 6.5.1-candidate 6.x-candidate DBaaS kubernetes releasenote triaged 6.0.5-candidate DBaaS approved-for-6.5.1 kubernetes releasenote triaged
            michael.blow Michael Blow made changes -
            Remote Link This issue links to "AsterixDB Gerrit Review (Web Link)" [ 19223 ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Michael Blow [ michael.blow ] Arunkumar Senthilnathan [ arunkumar ]
            arunkumar Arunkumar Senthilnathan made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            till Till Westmann made changes -
            Labels 6.0.5-candidate DBaaS approved-for-6.5.1 kubernetes releasenote triaged 6.0.6-candidate DBaaS approved-for-6.5.1 kubernetes releasenote triaged

            People

              arunkumar Arunkumar Senthilnathan
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty