Major Description
(I have reproduced it on release/7.2.5.xml, Trinity.xml (thus 7.6.3), and morpheus/master, thus only marking those as affected versions)
Target setting when client cert is "enabled":
This is from node 0 creating a remote cluster reference to the target (node1).
We can see that the root CA + client cert + client key are provided (truncated and redacted).
2024-08-13T14:55:28.969-07:00 INFO GOXDCR.AdminPort: doCreateRemoteClusterRequest req=&
|
|
|
{POST /pools/default/remoteClusters HTTP/1.1 1 1 map[Accept:[*/*] Authorization:[Basic xxxxx] Cb-On-Behalf-Of:[<ud>QWRtaW5pc3RyYXRvcjphZG1pbg==</ud>] Content-Length:[4548] Content-Type:[application/x-www-form-urlencoded] User-Agent:[curl/8.7.1]] 0xc000242ac0 <nil> 4548 [] false 127.0.0.1:9000 map[] map[] <nil> map[] 127.0.0.1:61519 /pools/default/remoteClusters <nil> <nil> <nil> 0xc0000b94a0 <nil> [] map[]}
|
|
|
2024-08-13T14:55:28.971-07:00 INFO GOXDCR.AdminPort: Request params: justValidate=false, remoteClusterRef=id:remoteCluster/O3PWnZQR0JkWZbLY9yn8OTmrISFymTsPP0EqZ51z0dw=; uuid:; name:C2; hostName:127.0.0.1:9001; userName:; password:; secureType:full; certificate:[… 62]; clientCertificate:[60 … 62]; clientKey:xxxx; SanInCertificate:false; HttpAuthMech:Plain, revision:<nil>
|
2024-08-13T14:55:28.971-07:00 INFO GOXDCR.RemClusterSvc: Adding remote cluster with referenceId remoteCluster/O3PWnZQR0JkWZbLY9yn8OTmrISFymTsPP0EqZ51z0dw=
|
2024-08-13T14:55:28.972-07:00 INFO GOXDCR.RemClusterSvc: C2 start of set host names and security, ref: isHTTPS:false,isEncEnabled:true,isFullEnc:true,isDNSSrv:false,host:[127.0.0.1:9001,],activeHost:[,]
|
2024-08-13T14:55:28.972-07:00 INFO GOXDCR.RemClusterSvc: C2 before querying default pools info refHostName=127.0.0.1:9001, refHttpsHostName=127.0.0.1:9001
|
2024-08-13T14:55:28.974-07:00 ERRO GOXDCR.RemClusterSvc: TLS handshake failed when connecting to 127.0.0.1:9001, err=tls: first record does not look like a TLS handshake
|
2024-08-13T14:55:28.974-07:00 ERRO GOXDCR.RemClusterSvc: Failed to get client for request, err=tls: first record does not look like a TLS handshake, req=map[Content-Type:[application/x-www-form-urlencoded] User-Agent:[couchbase-goxdcr/0.0.0]]
|
2024-08-13T14:55:29.061-07:00 INFO GOXDCR.RemClusterSvc: C2 after querying default pools info refHostName=127.0.0.1:9001, refHttpsHostName=127.0.0.1:19001
|
2024-08-13T14:55:29.061-07:00 INFO GOXDCR.RemClusterSvc: C2 nodelist after querying pools default info useExternal: false, nodelist: [[127.0.0.1:9001 127.0.0.1:19001]]
|
2024-08-13T14:55:29.061-07:00 INFO GOXDCR.RemClusterSvc: Set refHttpHostName=127.0.0.1:9001, refHttpsHostName=127.0.0.1:19001, SANInCertificate=true HttpAuthMech=Https for remote cluster reference remoteCluster/O3PWnZQR0JkWZbLY9yn8OTmrISFymTsPP0EqZ51z0dw=
|
2024-08-13T14:55:29.061-07:00 INFO GOXDCR.RemClusterSvc: C2 end of set host names and security
|
2024-08-13T14:55:29.061-07:00 INFO GOXDCR.RemClusterSvc: C2 after set hostname and security, ref: isHTTPS:true,isEncEnabled:true,isFullEnc:true,isDNSSrv:false,host:[127.0.0.1:9001,127.0.0.1:19001],activeHost:[127.0.0.1:9001,127.0.0.1:19001]
|
2024-08-13T14:55:29.081-07:00 INFO GOXDCR.RemClusterSvc: Result from validate remote cluster call: err=<nil>, statusCode=200. time taken=19.228217ms
|
Any logs contacting port 9001 can be ignored as XDCR will attempt both secure or nonSecure port in parallel and anticipate that one of them will speak TLS.
The key takeaway here is that the target ns_server was able to authenticate the REST call to the pools/default endpoint and returned a valid result with 200 status code. The call happens here:
https://github.com/couchbase/goxdcr/blob/7ea78c449e55042d43f6f97b9f6950278cfc59a9/metadata_svc/remote_cluster_service.go#L3175
The success of such a call will allow a XDCR remote cluster reference to be created.
Target setting when client cert is "mandatory":
The same exercise is done on the cluster but this time with "Mandatory" setting for client cert enablement.
From the logs, I confirmed that we're using client cert and key and not username/pw:
NEIL DEBUG using https on addr 127.0.0.1:19001 with username password cert len 1134 client cert len 1288 client key len 1703
|
The error code returned from ns_server would be 401 and XDCR then returns a more helpful error to the user:
https://github.com/couchbase/goxdcr/blob/7ea78c449e55042d43f6f97b9f6950278cfc59a9/utils/utils.go#L3035
This leads to the error being returned to the user:
NEIL DEBUG bgErr Received unauthorized error from target. Please double check user credentials. Since client certificate is being used, please ensure that target is version 5.5 and up and has client certificate authentication setting set to "enable" or "mandatory".
|
From node 1's http_access_internal.log, we see 401 returned:
127.0.0.1 - - [13/Aug/2024:15:29:25 -0700] "GET /pools/default HTTP/1.1" 401 0 - "couchbase-goxdcr/0.0.0" 1
|
I've done wireshark capture on port 19001 and attached them along with the node and client keys for both enabled (successful) and mandatory(401 returned) scenarios.
Attachments
Issue Links
- relates to
-
MB-63001 Client certificate authentication for cluster admin failing in 7.2.6/7.6.3
-
- Closed
-