Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54580

[BP 7.0.X] - [XDCR] Inter Cluster XDCR failing for 7.1.2 in Capella

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Won't Do
    • 7.0.0, 7.0.3, 7.0.2, 7.0.1, 7.0.4, 7.0.5
    • None
    • XDCR
    • Capella
    • Untriaged
    • Linux x86_64
    • 1
    • Yes
    • Critical

    Description

      Overview

      Inter-cluster XDCR is not working anymore where remote cluster creation fails when we pass the destination cluster’s SRV address as remote hostname and Capella’s CA cert.

      Steps to reproduce

      • Login to sbx-8 and create 2 AWS clusters (can also try with GCP)
      • Import a bucket in each cluster
      • Initiate inter-cluster XDCR
      • The XDCR will fail at create remote cluster step due to invalid IP SAN error in the destination cluster's node certificate.

      Details

       

      We had added this IP SAN due to MB-47901 : https://github.com/couchbasecloud/couchbase-cloud/blob/cd6141a8ac48342c178e41a5f5360ae948c9cff1/internal/clusters/certificates/service/manager.go#L383

      Before removing the IP, we got this error message during inter-cluster XDCR setup at the create remote cluster step:

      failed to create remote cluster:{"_":"Failed on calling host=10.0.16.86:18091, path=/pools, err=x509: certificate is valid for 127.0.0.1, not 10.0.16.86, statusCode=0"} 2: received error HTTP response: 400

       

      After removing the IP, we get this error:

      {"_":"Failed on calling host=10.11.4.76:18091, path=/pools, err=x509: cannot validate certificate for 10.11.4.76 because it doesn't contain any IP SANs, statusCode=0"}

       

      I tried to use the same CA cert and manually create remote cluster on the source cluster via the server UI console.
      I added:

      • hostname of remote cluster as the SRV record of the destination cluster.
      • CA cert in the TLS authentication CA cert input box
      • Username and password with full-admin role.

      I faced the same error message on the server UI. This helped us rule out requests and payload sent by Capella control plane to the server.

       

      VPC Peering

      • It is important to note that we initiate VPC peering for inter-cluster XDCR. If VPC peering is successful and we try to resolve the DNS address of any node in the destination cluster from a node in the source cluster, the address will always resolve to a private IP address.
      • The IP address mentioned in the above error messages belongs to a node in the destination cluster who's SRV record was stated as the hostname during create remote cluster step.

      Sample Cluster Certificate

      Certificate:
      Data:
      Version: 3 (0x2)
      Serial Number:
      87:c8:80:65:bb:78:4d:1e:68:56:ed:f5:20:78:3e:cc
      Signature Algorithm: sha512WithRSAEncryption
      Issuer: O=Couchbase, OU=Cloud, ST=California, L=Santa Clara
      Validity
      Not Before: Oct 23 07:20:19 2022 GMT
      Not After : Oct 23 08:20:19 2023 GMT
      Subject: C=US, ST=CA, O=Couchbase, OU=Cloud, CN=*.dfjaprjfetxdhu.sandbox.nonprod-project-avengers.com
      Subject Public Key Info:
      Public Key Algorithm: rsaEncryption
      Public-Key: (2048 bit)
      Modulus:
      00:a1:81:27:96:8c:30:ab:68:e3:79:a3:54:dd:dc:
      4e:82:2b:52:f4:20:f2:16:14:cd:2a:ce:22:86:f6:
      9b:35:da:0e:7f:0c:5e:23:9e:10:67:4e:c1:5e:01:
      b3:4a:ce:16:e7:23:fb:eb:89:05:97:a5:e6:3d:1d:
      40:bf:aa:eb:5b:83:fc:3e:e6:7e:ae:d0:91:dd:94:
      0f:18:e2:af:ab:94:9a:52:01:14:78:c9:b8:2c:44:
      65:88:f9:ad:0c:8d:c9:28:c0:8f:8a:41:75:5d:c5:
      c5:94:99:ef:97:e7:56:76:80:05:a9:b0:58:b8:27:
      54:9a:34:e7:93:6f:7f:b4:76:38:a9:e9:80:0f:b1:
      6e:99:db:4a:0d:48:2b:9e:a7:16:67:72:7c:88:81:
      bf:23:94:9a:10:31:86:d6:0d:cf:03:f9:12:f2:c8:
      fe:5f:cb:16:f9:c4:23:5a:7f:f3:3d:23:aa:b9:74:
      af:76:51:ce:f0:56:34:24:bf:58:7f:63:79:85:03:
      ef:2a:7c:81:79:ac:70:51:c0:21:fc:ed:d4:61:aa:
      cd:f9:05:95:2e:03:a9:79:d7:91:d5:1c:bd:d3:05:
      37:92:93:3c:c3:2c:75:a4:86:49:61:03:30:fd:00:
      4c:31:5b:97:66:97:8e:d1:2d:ca:d7:f1:71:8f:12:
      19:97
      Exponent: 65537 (0x10001)
      X509v3 extensions:
      X509v3 Subject Alternative Name:
      DNS:*.dfjaprjfetxdhu.sandbox.nonprod-project-avengers.com, DNS:localhost, IP Address:127.0.0.1
      X509v3 Basic Constraints:
      CA:FALSE
      X509v3 Authority Key Identifier:
      keyid:BD:29:52:36:B1:7C:FD:64:B1:19:A8:B8:9B:A8:9C:82:96:CF:7A:E1

      X509v3 Subject Key Identifier:
      3B:BE:3F:E0:8D:C6:80:E8:4C:6C:1B:6C:6F:74:F1:21:3A:45:77:B6
      X509v3 Key Usage: critical
      Digital Signature, Key Encipherment
      X509v3 Extended Key Usage:
      TLS Web Server Authentication, TLS Web Client Authentication
      Signature Algorithm: sha512WithRSAEncryption
      96:d0:25:92:f5:3d:d1:e1:b5:c6:23:12:9c:92:af:99:dd:e8:
      6d:ee:61:dc:ec:18:34:ce:b0:8f:cb:95:2e:96:98:74:dc:8c:
      2a:6d:45:3d:a0:97:b2:71:01:f3:7a:c9:83:1f:f7:3e:01:3d:
      8c:24:52:d8:c9:ed:e1:74:6a:95:52:40:ab:21:37:dc:33:3b:
      5d:84:be:01:d7:a8:24:8a:12:56:41:d4:0e:5b:b6:ee:1f:00:
      29:b6:cc:1c:25:9d:45:2e:ad:24:3e:fe:45:ca:fe:59:56:7d:
      2a:16:77:8a:8a:6d:00:b6:38:56:66:b2:eb:40:2d:7b:d2:75:
      c4:be:8b:72:63:f5:19:2b:e2:79:f8:2b:1d:39:d6:5c:17:1e:
      e3:70:6b:d9:de:9f:21:a4:f7:35:54:92:73:90:8e:67:a3:bd:
      9b:34:0c:07:c3:45:e7:89:70:06:60:5d:e2:e1:dc:3b:5f:02:
      20:2d:c4:30:ef:bb:49:81:49:a8:89:a0:f9:18:64:5d:e8:1e:
      73:e8:9d:49:f6:a5:8d:ef:14:f2:bd:0f:93:8c:dd:7c:ad:5e:
      e8:c9:02:05:88:72:99:21:b7:da:25:94:68:a7:19:35:6d:50:
      9b:70:88:99:57:84:c8:6b:b8:30:08:1a:60:32:33:94:8b:16:
      56:9c:af:9d
      ----BEGIN CERTIFICATE----
      MIIEKTCCAxGgAwIBAgIRAIfIgGW7eE0eaFbt9SB4PswwDQYJKoZIhvcNAQENBQAw
      TzESMBAGA1UECgwJQ291Y2hiYXNlMQ4wDAYDVQQLDAVDbG91ZDETMBEGA1UECAwK
      Q2FsaWZvcm5pYTEUMBIGA1UEBwwLU2FudGEgQ2xhcmEwHhcNMjIxMDIzMDcyMDE5
      WhcNMjMxMDIzMDgyMDE5WjB+MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQ
      BgNVBAoTCUNvdWNoYmFzZTEOMAwGA1UECxMFQ2xvdWQxPjA8BgNVBAMMNSouZGZq
      YXByamZldHhkaHUuc2FuZGJveC5ub25wcm9kLXByb2plY3QtYXZlbmdlcnMuY29t
      MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoYEnlowwq2jjeaNU3dxO
      gitS9CDyFhTNKs4ihvabNdoOfwxeI54QZ07BXgGzSs4W5yP764kFl6XmPR1Av6rr
      W4P8PuZ+rtCR3ZQPGOKvq5SaUgEUeMm4LERliPmtDI3JKMCPikF1XcXFlJnvl+dW
      doAFqbBYuCdUmjTnk29/tHY4qemAD7FumdtKDUgrnqcWZ3J8iIG/I5SaEDGG1g3P
      A/kS8sj+X8sW+cQjWn/zPSOquXSvdlHO8FY0JL9Yf2N5hQPvKnyBeaxwUcAh/O3U
      YarN+QWVLgOpedeR1Ry90wU3kpM8wyx1pIZJYQMw/QBMMVuXZpeO0S3K1/FxjxIZ
      lwIDAQABo4HQMIHNMFEGA1UdEQRKMEiCNSouZGZqYXByamZldHhkaHUuc2FuZGJv
      eC5ub25wcm9kLXByb2plY3QtYXZlbmdlcnMuY29tgglsb2NhbGhvc3SHBH8AAAEw
      CQYDVR0TBAIwADAfBgNVHSMEGDAWgBS9KVI2sXz9ZLEZqLibqJyCls964TAdBgNV
      HQ4EFgQUO74/4I3GgOhMbBtsb3TxITpFd7YwDgYDVR0PAQH/BAQDAgWgMB0GA1Ud
      JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0BAQ0FAAOCAQEAltAl
      kvU90eG1xiMSnJKvmd3obe5h3OwYNM6wj8uVLpaYdNyMKm1FPaCXsnEB83rJgx/3
      PgE9jCRS2Mnt4XRqlVJAqyE33DM7XYS+AdeoJIoSVkHUDlu27h8AKbbMHCWdRS6t
      JD7+Rcr+WVZ9KhZ3ioptALY4Vmay60Ate9J1xL6LcmP1GSviefgrHTnWXBce43Br
      2d6fIaT3NVSSc5COZ6O9mzQMB8NF54lwBmBd4uHcO18CIC3EMO+7SYFJqImg+Rhk
      Xegec+idSfalje8U8r0Pk4zdfK1e6MkCBYhymSG32iWUaKcZNW1Qm3CImVeEyGu4
      MAgaYDIzlIsWVpyvnQ==
      ----END CERTIFICATE----

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            neil.huang Neil Huang added a comment -

            No 7.0.6 vehicle as of now.

            neil.huang Neil Huang added a comment - No 7.0.6 vehicle as of now.

            People

              neil.huang Neil Huang
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty