Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37458

[CX] Hostnames for public addresses should be resolved on the client side

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Is this a Regression?:
      Unknown
    • Sprint:
      CX Sprint 181

      Description

      Observed in post-mortem logs of CBSE-7813, it appears the issue is that the the publicAddress of an azure node (e.g. node A) is resolving to 0.0.0.0 or localhost. When another node (e.g. node B) attempts to contact node A, it actually contacts itself, since it uses the resolved form of the public address, which is pointing to localhost.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            michael.blow Michael Blow added a comment -

            The fix was incomplete; another change is necessary to fully address the issue.

            Show
            michael.blow Michael Blow added a comment - The fix was incomplete; another change is necessary to fully address the issue.
            Show
            michael.blow Michael Blow added a comment - https://github.com/couchbase/asterixdb/commit/d0710cefe032017800887fe68cbbb8a25a8f28a1
            Hide
            michael.blow Michael Blow added a comment - - edited

            Analytics cluster of size > 1 will not work on Azure nodes without this fix, assuming the Couchbase "best-practices" instructions were followed when configuring the cluster. The second nodes will not be able to successfully join the cluster.

            The issue is caused by resolving the name of each node on the node itself, instead of resolving the names to addresses on the caller side. The best-practices document instructs users to assign the public DNS node name to localhost in the hosts file, causing the callers to erroneously attempt to contact nodes on their own loopback interface.

            A workaround is to remove the mapping of the node name to localhost on each node's hosts file. This issue is fixed in 6.0.4, 6.5.1, and Chesire-Cat.

            Show
            michael.blow Michael Blow added a comment - - edited Analytics cluster of size > 1 will not work on Azure nodes without this fix, assuming the Couchbase "best-practices" instructions were followed when configuring the cluster. The second nodes will not be able to successfully join the cluster. The issue is caused by resolving the name of each node on the node itself, instead of resolving the names to addresses on the caller side. The best-practices document instructs users to assign the public DNS node name to localhost in the hosts file, causing the callers to erroneously attempt to contact nodes on their own loopback interface. A workaround is to remove the mapping of the node name to localhost on each node's hosts file. This issue is fixed in 6.0.4, 6.5.1, and Chesire-Cat.
            Hide
            mihir.kamdar Mihir Kamdar added a comment -

            Verified on 6.0.4-3077 on Azure. Issue is resolved. Able to add another analytics node to the cluster. With 6.0.3, rebalance in for a new analytics node failed.

            Show
            mihir.kamdar Mihir Kamdar added a comment - Verified on 6.0.4-3077 on Azure. Issue is resolved. Able to add another analytics node to the cluster. With 6.0.3, rebalance in for a new analytics node failed.
            Hide
            mihir.kamdar Mihir Kamdar added a comment -

            Cannot verify on 6.5.1 because of MB-38655

            Show
            mihir.kamdar Mihir Kamdar added a comment - Cannot verify on 6.5.1 because of MB-38655
            Hide
            mihir.kamdar Mihir Kamdar added a comment -

            Verified on 6.5.1-6296 and 7.0.0-1650

            Show
            mihir.kamdar Mihir Kamdar added a comment - Verified on 6.5.1-6296 and 7.0.0-1650

              People

              Assignee:
              mihir.kamdar Mihir Kamdar
              Reporter:
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty