[BP 7.1.X] - [XDCR] Inter Cluster XDCR failing in Server 7.1.2 and Capella

Overview

Inter-cluster XDCR is not working anymore where remote cluster creation fails when we pass the destination cluster’s SRV address as remote hostname and Capella’s CA cert.

Steps to reproduce

• Login to sbx-8 and create 2 AWS clusters (can also try with GCP)
• Import a bucket in each cluster
• Initiate inter-cluster XDCR
• The XDCR will fail at create remote cluster step due to invalid IP SAN error in the destination cluster's node certificate.

Details

• We create a CA certificate that is common for all clusters in Capella.
• For all nodes in a single cluster, we create a client cert per node (but this client cert is common for all nodes in that cluster) Ref: https://github.com/couchbasecloud/couchbase-cloud/blob/main/internal/clusters/certificates/service/manager.go#L113
• We generate the above cluster cert here: https://github.com/couchbasecloud/couchbase-cloud/blob/cd6141a8ac48342c178e41a5f5360ae948c9cff1/internal/clusters/certificates/service/manager.go#L357
• We use the wildcard DNS entry as the SAN while creating the certificate: https://github.com/couchbasecloud/couchbase-cloud/blob/cd6141a8ac48342c178e41a5f5360ae948c9cff1/internal/clusters/certificates/service/manager.go#L378 This way, each node's hostname can be auto-resolved to the wildcard SAN in the certificate during certificate validation.
• We create the CSR here: https://github.com/couchbasecloud/couchbase-cloud/blob/cd6141a8ac48342c178e41a5f5360ae948c9cff1/internal/clusters/certificates/service/manager.go#L423
• We sign the cluster certificate with the CA cert here: https://github.com/couchbasecloud/couchbase-cloud/blob/cd6141a8ac48342c178e41a5f5360ae948c9cff1/internal/clusters/certificates/service/manager.go#L457

We had added this IP SAN due to MB-47901 : https://github.com/couchbasecloud/couchbase-cloud/blob/cd6141a8ac48342c178e41a5f5360ae948c9cff1/internal/clusters/certificates/service/manager.go#L383

Before removing the IP, we got this error message during inter-cluster XDCR setup at the create remote cluster step:

failed to create remote cluster:{"_":"Failed on calling host=10.0.16.86:18091, path=/pools, err=x509: certificate is valid for 127.0.0.1, not 10.0.16.86, statusCode=0"} 2: received error HTTP response: 400

After removing the IP, we get this error:

I tried to use the same CA cert and manually create remote cluster on the source cluster via the server UI console.
I added:

• hostname of remote cluster as the SRV record of the destination cluster.
• CA cert in the TLS authentication CA cert input box
• Username and password with full-admin role.

I faced the same error message on the server UI. This helped us rule out requests and payload sent by Capella control plane to the server.

VPC Peering

• It is important to note that we initiate VPC peering for inter-cluster XDCR. If VPC peering is successful and we try to resolve the DNS address of any node in the destination cluster from a node in the source cluster, the address will always resolve to a private IP address.
• The IP address mentioned in the above error messages belongs to a node in the destination cluster who's SRV record was stated as the hostname during create remote cluster step.

Sample Cluster Certificate

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
87:c8:80:65:bb:78:4d:1e:68:56:ed:f5:20:78:3e:cc
Signature Algorithm: sha512WithRSAEncryption
Issuer: O=Couchbase, OU=Cloud, ST=California, L=Santa Clara
Validity
Not Before: Oct 23 07:20:19 2022 GMT
Not After : Oct 23 08:20:19 2023 GMT
Subject: C=US, ST=CA, O=Couchbase, OU=Cloud, CN=*.dfjaprjfetxdhu.sandbox.nonprod-project-avengers.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:a1:81:27:96:8c:30:ab:68:e3:79:a3:54:dd:dc:
4e:82:2b:52:f4:20:f2:16:14:cd:2a:ce:22:86:f6:
9b:35:da:0e:7f:0c:5e:23:9e:10:67:4e:c1:5e:01:
b3:4a:ce:16:e7:23:fb:eb:89:05:97:a5:e6:3d:1d:
40:bf:aa:eb:5b:83:fc:3e:e6:7e:ae:d0:91:dd:94:
0f:18:e2:af:ab:94:9a:52:01:14:78:c9:b8:2c:44:
65:88:f9:ad:0c:8d:c9:28:c0:8f:8a:41:75:5d:c5:
c5:94:99:ef:97:e7:56:76:80:05:a9:b0:58:b8:27:
54:9a:34:e7:93:6f:7f:b4:76:38:a9:e9:80:0f:b1:
6e:99:db:4a:0d:48:2b:9e:a7:16:67:72:7c:88:81:
bf:23:94:9a:10:31:86:d6:0d:cf:03:f9:12:f2:c8:
fe:5f:cb:16:f9:c4:23:5a:7f:f3:3d:23:aa:b9:74:
af:76:51:ce:f0:56:34:24:bf:58:7f:63:79:85:03:
ef:2a:7c:81:79:ac:70:51:c0:21:fc:ed:d4:61:aa:
cd:f9:05:95:2e:03:a9:79:d7:91:d5:1c:bd:d3:05:
37:92:93:3c:c3:2c:75:a4:86:49:61:03:30:fd:00:
4c:31:5b:97:66:97:8e:d1:2d:ca:d7:f1:71:8f:12:
19:97
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:*.dfjaprjfetxdhu.sandbox.nonprod-project-avengers.com, DNS:localhost, IP Address:127.0.0.1
X509v3 Basic Constraints:
CA:FALSE
X509v3 Authority Key Identifier:
keyid:BD:29:52:36:B1:7C:FD:64:B1:19:A8:B8:9B:A8:9C:82:96:CF:7A:E1

X509v3 Subject Key Identifier:
3B:BE:3F:E0:8D:C6:80:E8:4C:6C:1B:6C:6F:74:F1:21:3A:45:77:B6
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
Signature Algorithm: sha512WithRSAEncryption
96:d0:25:92:f5:3d:d1:e1:b5:c6:23:12:9c:92:af:99:dd:e8:
6d:ee:61:dc:ec:18:34:ce:b0:8f:cb:95:2e:96:98:74:dc:8c:
2a:6d:45:3d:a0:97:b2:71:01:f3:7a:c9:83:1f:f7:3e:01:3d:
8c:24:52:d8:c9:ed:e1:74:6a:95:52:40:ab:21:37:dc:33:3b:
5d:84:be:01:d7:a8:24:8a:12:56:41:d4:0e:5b:b6:ee:1f:00:
29:b6:cc:1c:25:9d:45:2e:ad:24:3e:fe:45:ca:fe:59:56:7d:
2a:16:77:8a:8a:6d:00:b6:38:56:66:b2:eb:40:2d:7b:d2:75:
c4:be:8b:72:63:f5:19:2b:e2:79:f8:2b:1d:39:d6:5c:17:1e:
e3:70:6b:d9:de:9f:21:a4:f7:35:54:92:73:90:8e:67:a3:bd:
9b:34:0c:07:c3:45:e7:89:70:06:60:5d:e2:e1:dc:3b:5f:02:
20:2d:c4:30:ef:bb:49:81:49:a8:89:a0:f9:18:64:5d:e8:1e:
73:e8:9d:49:f6:a5:8d:ef:14:f2:bd:0f:93:8c:dd:7c:ad:5e:
e8:c9:02:05:88:72:99:21:b7:da:25:94:68:a7:19:35:6d:50:
9b:70:88:99:57:84:c8:6b:b8:30:08:1a:60:32:33:94:8b:16:
56:9c:af:9d
----BEGIN CERTIFICATE----
MIIEKTCCAxGgAwIBAgIRAIfIgGW7eE0eaFbt9SB4PswwDQYJKoZIhvcNAQENBQAw
TzESMBAGA1UECgwJQ291Y2hiYXNlMQ4wDAYDVQQLDAVDbG91ZDETMBEGA1UECAwK
Q2FsaWZvcm5pYTEUMBIGA1UEBwwLU2FudGEgQ2xhcmEwHhcNMjIxMDIzMDcyMDE5
WhcNMjMxMDIzMDgyMDE5WjB+MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQ
BgNVBAoTCUNvdWNoYmFzZTEOMAwGA1UECxMFQ2xvdWQxPjA8BgNVBAMMNSouZGZq
YXByamZldHhkaHUuc2FuZGJveC5ub25wcm9kLXByb2plY3QtYXZlbmdlcnMuY29t
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAoYEnlowwq2jjeaNU3dxO
gitS9CDyFhTNKs4ihvabNdoOfwxeI54QZ07BXgGzSs4W5yP764kFl6XmPR1Av6rr
W4P8PuZ+rtCR3ZQPGOKvq5SaUgEUeMm4LERliPmtDI3JKMCPikF1XcXFlJnvl+dW
doAFqbBYuCdUmjTnk29/tHY4qemAD7FumdtKDUgrnqcWZ3J8iIG/I5SaEDGG1g3P
A/kS8sj+X8sW+cQjWn/zPSOquXSvdlHO8FY0JL9Yf2N5hQPvKnyBeaxwUcAh/O3U
YarN+QWVLgOpedeR1Ry90wU3kpM8wyx1pIZJYQMw/QBMMVuXZpeO0S3K1/FxjxIZ
lwIDAQABo4HQMIHNMFEGA1UdEQRKMEiCNSouZGZqYXByamZldHhkaHUuc2FuZGJv
eC5ub25wcm9kLXByb2plY3QtYXZlbmdlcnMuY29tgglsb2NhbGhvc3SHBH8AAAEw
CQYDVR0TBAIwADAfBgNVHSMEGDAWgBS9KVI2sXz9ZLEZqLibqJyCls964TAdBgNV
HQ4EFgQUO74/4I3GgOhMbBtsb3TxITpFd7YwDgYDVR0PAQH/BAQDAgWgMB0GA1Ud
JQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjANBgkqhkiG9w0BAQ0FAAOCAQEAltAl
kvU90eG1xiMSnJKvmd3obe5h3OwYNM6wj8uVLpaYdNyMKm1FPaCXsnEB83rJgx/3
PgE9jCRS2Mnt4XRqlVJAqyE33DM7XYS+AdeoJIoSVkHUDlu27h8AKbbMHCWdRS6t
JD7+Rcr+WVZ9KhZ3ioptALY4Vmay60Ate9J1xL6LcmP1GSviefgrHTnWXBce43Br
2d6fIaT3NVSSc5COZ6O9mzQMB8NF54lwBmBd4uHcO18CIC3EMO+7SYFJqImg+Rhk
Xegec+idSfalje8U8r0Pk4zdfK1e6MkCBYhymSG32iWUaKcZNW1Qm3CImVeEyGu4
MAgaYDIzlIsWVpyvnQ==
----END CERTIFICATE----