Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40358

TLS with client certificate for external link is not working.

    XMLWordPrintable

    Details

      Description

      Steps to reproduce-

      1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

      2. generate certificates root, node and client certificates for both the cluster.

      3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

      4. link creation failed.

      Error when executing from postman-
      CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
       
      Error when executing using curl
      curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/long_chain172.16.1.174.pem)”  --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
      curl: option -----END: is unknown
      curl: try 'curl --help' or 'curl --manual' for more information
       
      curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/ca.pem)”  --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
      curl: option -----END: is unknown
      curl: try 'curl --help' or 'curl --manual' for more information
      

      Have verified that the certificates that were created are working.

      curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key  https://10.112.200.104:18091/pools/default
      *   Trying 10.112.200.104...
      * TCP_NODELAY set
      * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
      * ALPN, offering h2
      * ALPN, offering http/1.1
      * successfully set certificate verify locations:
      *   CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem
        CApath: none
      * TLSv1.2 (OUT), TLS handshake, Client hello (1):
      * TLSv1.2 (IN), TLS handshake, Server hello (2):
      * TLSv1.2 (IN), TLS handshake, Certificate (11):
      * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
      * TLSv1.2 (IN), TLS handshake, Request CERT (13):
      * TLSv1.2 (IN), TLS handshake, Server finished (14):
      * TLSv1.2 (OUT), TLS handshake, Certificate (11):
      * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
      * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
      * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
      * TLSv1.2 (OUT), TLS handshake, Finished (20):
      * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
      * TLSv1.2 (IN), TLS handshake, Finished (20):
      * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
      * ALPN, server did not agree to a protocol
      * Server certificate:
      *  subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
      *  start date: Jul  9 04:22:00 2020 GMT
      *  expire date: Jul  9 04:22:00 2021 GMT
      *  subjectAltName: host "10.112.200.104" matched cert's IP address!
      *  issuer: C=UA; O=My Company; CN=My Company Intermediate CA
      *  SSL certificate verify ok.
      > GET /pools/default HTTP/1.1
      > Host: 10.112.200.104:18091
      > User-Agent: curl/7.64.1
      > Accept: */*
      > 
      < HTTP/1.1 200 OK
      < X-XSS-Protection: 1; mode=block
      < X-Permitted-Cross-Domain-Policies: none
      < X-Frame-Options: DENY
      < X-Content-Type-Options: nosniff
      < Server: Couchbase Server
      < Pragma: no-cache
      < Expires: Thu, 01 Jan 1970 00:00:00 GMT
      < Date: Thu, 09 Jul 2020 04:32:12 GMT
      < Content-Type: application/json
      < Content-Length: 4181
      < Cache-Control: no-cache,no-store,must-revalidate
      < 
      {"name":"default","nodes":[{"systemStats":{"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]}],"buckets":{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":{"percentage":30}},"tasks":{"uri":"/pools/default/tasks?v=35395949"},"counters":{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":{"total":198285* Connection #0 to host 10.112.200.104 left intact
      72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
      

      Have also verified that the above API endpoint does not works without authentication:

      curl -v  http://10.112.200.104:8091/pools/default
      *   Trying 10.112.200.104...
      * TCP_NODELAY set
      * Connected to 10.112.200.104 (10.112.200.104) port 8091 (#0)
      > GET /pools/default HTTP/1.1
      > Host: 10.112.200.104:8091
      > User-Agent: curl/7.64.1
      > Accept: */*
      > 
      < HTTP/1.1 401 Unauthorized
      < X-XSS-Protection: 1; mode=block
      < X-Permitted-Cross-Domain-Policies: none
      < X-Frame-Options: DENY
      < X-Content-Type-Options: nosniff
      < WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"
      < Server: Couchbase Server
      < Pragma: no-cache
      < Expires: Thu, 01 Jan 1970 00:00:00 GMT
      < Date: Thu, 09 Jul 2020 05:00:22 GMT
      < Content-Length: 0
      < Cache-Control: no-cache,no-store,must-revalidate
      < 
      * Connection #0 to host 10.112.200.104 left intact
      * Closing connection 0
      

      Attaching all the certificates that i generated.

      Node certificates -
      10.112.200.104.csr ,10.112.200.104.key , 10.112.200.104.pem

      Client certificates -
      172.16.1.174.csr, 172.16.1.174.key, 172.16.1.174.pem

      root certificates-
      ca.key, ca.pem

      Intermediate certificates-
      int.csr, int.key, int.pem, intermediateCA.srl

      other certificates-
      long_chain10.112.200.104.pem, long_chain172.16.1.174.pem, root.crt, rootCA.srl

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            michael.blow Michael Blow added a comment -

            Notice the curl commands are failing

            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information
            

            it looks like you have the wrong quotation marks in use at least a few places in your commands. They need to be " not ” or ”.

            As an alternative, please try the CLI.

            Show
            michael.blow Michael Blow added a comment - Notice the curl commands are failing curl: option -----END: is unknown curl: try 'curl --help' or 'curl --manual' for more information it looks like you have the wrong quotation marks in use at least a few places in your commands. They need to be " not ” or ”. As an alternative, please try the CLI.
            Hide
            umang.agrawal Umang added a comment -

            Getting error while creating remote link using TLS with client certificate, using both curl and couchbase-cli.

            Using CURL-

            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts46C1/ca.pem)"  --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts46C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts46C1/172.16.1.174.key)"
             
            Note: Unnecessary use of -X or --request, POST is already inferred.
            *   Trying 10.112.200.103...
            * TCP_NODELAY set
            * Connected to 10.112.200.103 (10.112.200.103) port 8095 (#0)
            * Server auth using Basic with user 'Administrator'
            > POST /analytics/link HTTP/1.1
            > Host: 10.112.200.103:8095
            > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
            > User-Agent: curl/7.64.1
            > Accept: */*
            > Content-Length: 2870
            > Content-Type: application/x-www-form-urlencoded
            > Expect: 100-continue
            > 
            < HTTP/1.1 100 Continue
            * We are completely uploaded and fine
            < HTTP/1.1 400 Bad Request
            < connection: keep-alive
            < content-type: text/plain; charset=UTF-8
            < content-length: 86
            < 
            CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
            * Connection #0 to host 10.112.200.103 left intact
            * Closing connection 0
            
            

            Show
            umang.agrawal Umang added a comment - Getting error while creating remote link using TLS with client certificate, using both curl and couchbase-cli. Using CURL- curl -v -u Administrator:password -X POST http: //10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts46C1/ca.pem)" --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts46C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts46C1/172.16.1.174.key)"   Note: Unnecessary use of -X or --request, POST is already inferred. * Trying 10.112 . 200.103 ... * TCP_NODELAY set * Connected to 10.112 . 200.103 ( 10.112 . 200.103 ) port 8095 (# 0 ) * Server auth using Basic with user 'Administrator' > POST /analytics/link HTTP/ 1.1 > Host: 10.112 . 200.103 : 8095 > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA== > User-Agent: curl/ 7.64 . 1 > Accept: */* > Content-Length: 2870 > Content-Type: application/x-www-form-urlencoded > Expect: 100 - continue > < HTTP/ 1.1 100 Continue * We are completely uploaded and fine < HTTP/ 1.1 400 Bad Request < connection: keep-alive < content-type: text/plain; charset=UTF- 8 < content-length: 86 < CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord * Connection # 0 to host 10.112 . 200.103 left intact * Closing connection 0
            Hide
            umang.agrawal Umang added a comment -

            Attaching new certs and server logs. newcerts45C1.zip

            Show
            umang.agrawal Umang added a comment - Attaching new certs and server logs. newcerts45C1.zip
            Hide
            michael.blow Michael Blow added a comment -

            all the certs i am generating are generated using scripts that we have for automation. only the names change for the files.

            Can you provide these scripts? It's hard to use these certificates as my hosts do not have the same IP addresses

            Show
            michael.blow Michael Blow added a comment - all the certs i am generating are generated using scripts that we have for automation. only the names change for the files. Can you provide these scripts? It's hard to use these certificates as my hosts do not have the same IP addresses
            Hide
            umang.agrawal Umang added a comment -

            You can find script to generate certificate here -
            https://github.com/couchbaselabs/TAF/blob/master/couchbase_utils/security_utils/x509main.py

            You can check out usage of the above script here -
            https://github.com/couchbaselabs/TAF/blob/master/pytests/cbas/cbas_external_links_CB_cluster.py
            Line 82-125 in the above file.

            Show
            umang.agrawal Umang added a comment - You can find script to generate certificate here - https://github.com/couchbaselabs/TAF/blob/master/couchbase_utils/security_utils/x509main.py You can check out usage of the above script here - https://github.com/couchbaselabs/TAF/blob/master/pytests/cbas/cbas_external_links_CB_cluster.py Line 82-125 in the above file.
            Hide
            michael.blow Michael Blow added a comment -

            Hi Umang,

            As these are client certificates being created with an intermediate authority, the client certificate being supplied to the create or alter link API needs to have the intermediate certificate appended to it, as is described in step 10 of Client Access: Intermediate-Certificate Authorization

            In the example above, you would need to

            cat 172.16.1.174.pem int.pem > 172.16.1.174_chain.pem
            

            , and use 172.16.1.174_chain.pem for the create / alter link

            Show
            michael.blow Michael Blow added a comment - Hi Umang , As these are client certificates being created with an intermediate authority, the client certificate being supplied to the create or alter link API needs to have the intermediate certificate appended to it, as is described in step 10 of Client Access: Intermediate-Certificate Authorization In the example above, you would need to cat 172.16 . 1.174 .pem int .pem > 172.16 . 1 .174_chain.pem , and use 172.16.1.174_chain.pem for the create / alter link
            Hide
            ritam.sharma Ritam Sharma added a comment - - edited

            Michael Blow = long_chain172.16.1.174.pem - this is the chain cert for

            /tmp/newcerts3/172.16.1.174.pem /tmp/newcerts3/int.pem /tmp/newcerts3/ca.pem > /tmp/newcerts3/long_chain172.16.1.174.pem

            Above was tested with both ca and chain certs.

            Umang - can you please update ticket with chain cert.

            Show
            ritam.sharma Ritam Sharma added a comment - - edited Michael Blow = long_chain172.16.1.174.pem - this is the chain cert for /tmp/newcerts3/172.16.1.174.pem /tmp/newcerts3/int.pem /tmp/newcerts3/ca.pem > /tmp/newcerts3/long_chain172.16.1.174.pem Above was tested with both ca and chain certs. Umang - can you please update ticket with chain cert.
            Hide
            umang.agrawal Umang added a comment -

            after appending intermediate cert in client cert, the link creation is working as expected.
            Verified with couchbase server build 6.6.0-7878

            Show
            umang.agrawal Umang added a comment - after appending intermediate cert in client cert, the link creation is working as expected. Verified with couchbase server build 6.6.0-7878
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.0-7885 contains cbas-core commit 36fe8fa with commit message:
            MB-40358: use intermediate certificate authority

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7885 contains cbas-core commit 36fe8fa with commit message: MB-40358 : use intermediate certificate authority
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-2640 contains cbas-core commit 36fe8fa with commit message:
            MB-40358: use intermediate certificate authority

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2640 contains cbas-core commit 36fe8fa with commit message: MB-40358 : use intermediate certificate authority
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit 36fe8fa with commit message:
            MB-40358: use intermediate certificate authority

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 36fe8fa with commit message: MB-40358 : use intermediate certificate authority

              People

              Assignee:
              umang.agrawal Umang
              Reporter:
              umang.agrawal Umang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty