Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40358

TLS with client certificate for external link is not working.

    XMLWordPrintable

    Details

      Description

      Steps to reproduce-

      1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

      2. generate certificates root, node and client certificates for both the cluster.

      3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

      4. link creation failed.

      Error when executing from postman-
      CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
       
      Error when executing using curl
      curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/long_chain172.16.1.174.pem)”  --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
      curl: option -----END: is unknown
      curl: try 'curl --help' or 'curl --manual' for more information
       
      curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/ca.pem)”  --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
      curl: option -----END: is unknown
      curl: try 'curl --help' or 'curl --manual' for more information
      

      Have verified that the certificates that were created are working.

      curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key  https://10.112.200.104:18091/pools/default
      *   Trying 10.112.200.104...
      * TCP_NODELAY set
      * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
      * ALPN, offering h2
      * ALPN, offering http/1.1
      * successfully set certificate verify locations:
      *   CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem
        CApath: none
      * TLSv1.2 (OUT), TLS handshake, Client hello (1):
      * TLSv1.2 (IN), TLS handshake, Server hello (2):
      * TLSv1.2 (IN), TLS handshake, Certificate (11):
      * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
      * TLSv1.2 (IN), TLS handshake, Request CERT (13):
      * TLSv1.2 (IN), TLS handshake, Server finished (14):
      * TLSv1.2 (OUT), TLS handshake, Certificate (11):
      * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
      * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
      * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
      * TLSv1.2 (OUT), TLS handshake, Finished (20):
      * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
      * TLSv1.2 (IN), TLS handshake, Finished (20):
      * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
      * ALPN, server did not agree to a protocol
      * Server certificate:
      *  subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
      *  start date: Jul  9 04:22:00 2020 GMT
      *  expire date: Jul  9 04:22:00 2021 GMT
      *  subjectAltName: host "10.112.200.104" matched cert's IP address!
      *  issuer: C=UA; O=My Company; CN=My Company Intermediate CA
      *  SSL certificate verify ok.
      > GET /pools/default HTTP/1.1
      > Host: 10.112.200.104:18091
      > User-Agent: curl/7.64.1
      > Accept: */*
      > 
      < HTTP/1.1 200 OK
      < X-XSS-Protection: 1; mode=block
      < X-Permitted-Cross-Domain-Policies: none
      < X-Frame-Options: DENY
      < X-Content-Type-Options: nosniff
      < Server: Couchbase Server
      < Pragma: no-cache
      < Expires: Thu, 01 Jan 1970 00:00:00 GMT
      < Date: Thu, 09 Jul 2020 04:32:12 GMT
      < Content-Type: application/json
      < Content-Length: 4181
      < Cache-Control: no-cache,no-store,must-revalidate
      < 
      {"name":"default","nodes":[{"systemStats":{"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]}],"buckets":{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":{"percentage":30}},"tasks":{"uri":"/pools/default/tasks?v=35395949"},"counters":{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":{"total":198285* Connection #0 to host 10.112.200.104 left intact
      72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
      

      Have also verified that the above API endpoint does not works without authentication:

      curl -v  http://10.112.200.104:8091/pools/default
      *   Trying 10.112.200.104...
      * TCP_NODELAY set
      * Connected to 10.112.200.104 (10.112.200.104) port 8091 (#0)
      > GET /pools/default HTTP/1.1
      > Host: 10.112.200.104:8091
      > User-Agent: curl/7.64.1
      > Accept: */*
      > 
      < HTTP/1.1 401 Unauthorized
      < X-XSS-Protection: 1; mode=block
      < X-Permitted-Cross-Domain-Policies: none
      < X-Frame-Options: DENY
      < X-Content-Type-Options: nosniff
      < WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"
      < Server: Couchbase Server
      < Pragma: no-cache
      < Expires: Thu, 01 Jan 1970 00:00:00 GMT
      < Date: Thu, 09 Jul 2020 05:00:22 GMT
      < Content-Length: 0
      < Cache-Control: no-cache,no-store,must-revalidate
      < 
      * Connection #0 to host 10.112.200.104 left intact
      * Closing connection 0
      

      Attaching all the certificates that i generated.

      Node certificates -
      10.112.200.104.csr ,10.112.200.104.key , 10.112.200.104.pem

      Client certificates -
      172.16.1.174.csr, 172.16.1.174.key, 172.16.1.174.pem

      root certificates-
      ca.key, ca.pem

      Intermediate certificates-
      int.csr, int.key, int.pem, intermediateCA.srl

      other certificates-
      long_chain10.112.200.104.pem, long_chain172.16.1.174.pem, root.crt, rootCA.srl

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            umang.agrawal Umang created issue -
            umang.agrawal Umang made changes -
            Field Original Value New Value
            Description Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}
            CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
            {code}
            Have verified that the certificates that were created are working.
            umang.agrawal Umang made changes -
            Description Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}
            CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
            {code}
            Have verified that the certificates that were created are working.
            Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
            {code}
            Have verified that the certificates that were created are working.
            {code:java}
            curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key https://10.112.200.104:18091/pools/default

            {code}
             * Trying 10.112.200.104...
             * TCP_NODELAY set
             * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
             * ALPN, offering h2 * ALPN, offering http/1.1
             * successfully set certificate verify locations:
             * CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem CApath: none
             * TLSv1.2 (OUT), TLS handshake, Client hello (1):
             * TLSv1.2 (IN), TLS handshake, Server hello (2):
             * TLSv1.2 (IN), TLS handshake, Certificate (11):
             * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
             * TLSv1.2 (IN), TLS handshake, Request CERT (13):
             * TLSv1.2 (IN), TLS handshake, Server finished (14):
             * TLSv1.2 (OUT), TLS handshake, Certificate (11):
             * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
             * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
             * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
             * TLSv1.2 (OUT), TLS handshake, Finished (20):
             * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
             * TLSv1.2 (IN), TLS handshake, Finished (20):
             * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
             * ALPN, server did not agree to a protocol
             * Server certificate: * subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
             * start date: Jul 9 04:22:00 2020 GMT
             * expire date: Jul 9 04:22:00 2021 GMT
             * subjectAltName: host "10.112.200.104" matched cert's IP address!
             * issuer: C=UA; O=My Company; CN=My Company Intermediate CA
             * SSL certificate verify ok.
             * > GET /pools/default HTTP/1.1
             * > Host: 10.112.200.104:18091
             * > User-Agent: curl/7.64.1
             * > Accept: */* >
             * < HTTP/1.1 200 OK
             * < X-XSS-Protection: 1; mode=block
             * < X-Permitted-Cross-Domain-Policies: none
             * < X-Frame-Options: DENY
             * < X-Content-Type-Options: nosniff
             * < Server: Couchbase Server
             * < Pragma: no-cache
             * < Expires: Thu, 01 Jan 1970 00:00:00 GMT
             * < Date: Thu, 09 Jul 2020 04:32:12 GMT
             * < Content-Type: application/json < Content-Length: 4181 < Cache-Control: no-cache,no-store,must-revalidate
             * < {"name":"default","nodes":[{"systemStats":
             * {"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":\{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[\{"afamily":"inet","nodeEncryption":false},\{"afamily":"inet6","nodeEncryption":false}]}],"buckets":\{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":\{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":\{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":\{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":\{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":\{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":\{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":\{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":\{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":\{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":\{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":\{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":\{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":\{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":\{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":\{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":\{"percentage":30}},"tasks":\{"uri":"/pools/default/tasks?v=35395949"},"counters":\{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":\{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":\{"total":198285* Connection #0 to host 10.112.200.104 left intact 72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
            umang.agrawal Umang made changes -
            Description Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord
            {code}
            Have verified that the certificates that were created are working.
            {code:java}
            curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key https://10.112.200.104:18091/pools/default

            {code}
             * Trying 10.112.200.104...
             * TCP_NODELAY set
             * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
             * ALPN, offering h2 * ALPN, offering http/1.1
             * successfully set certificate verify locations:
             * CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem CApath: none
             * TLSv1.2 (OUT), TLS handshake, Client hello (1):
             * TLSv1.2 (IN), TLS handshake, Server hello (2):
             * TLSv1.2 (IN), TLS handshake, Certificate (11):
             * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
             * TLSv1.2 (IN), TLS handshake, Request CERT (13):
             * TLSv1.2 (IN), TLS handshake, Server finished (14):
             * TLSv1.2 (OUT), TLS handshake, Certificate (11):
             * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
             * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
             * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
             * TLSv1.2 (OUT), TLS handshake, Finished (20):
             * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
             * TLSv1.2 (IN), TLS handshake, Finished (20):
             * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
             * ALPN, server did not agree to a protocol
             * Server certificate: * subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
             * start date: Jul 9 04:22:00 2020 GMT
             * expire date: Jul 9 04:22:00 2021 GMT
             * subjectAltName: host "10.112.200.104" matched cert's IP address!
             * issuer: C=UA; O=My Company; CN=My Company Intermediate CA
             * SSL certificate verify ok.
             * > GET /pools/default HTTP/1.1
             * > Host: 10.112.200.104:18091
             * > User-Agent: curl/7.64.1
             * > Accept: */* >
             * < HTTP/1.1 200 OK
             * < X-XSS-Protection: 1; mode=block
             * < X-Permitted-Cross-Domain-Policies: none
             * < X-Frame-Options: DENY
             * < X-Content-Type-Options: nosniff
             * < Server: Couchbase Server
             * < Pragma: no-cache
             * < Expires: Thu, 01 Jan 1970 00:00:00 GMT
             * < Date: Thu, 09 Jul 2020 04:32:12 GMT
             * < Content-Type: application/json < Content-Length: 4181 < Cache-Control: no-cache,no-store,must-revalidate
             * < {"name":"default","nodes":[{"systemStats":
             * {"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":\{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[\{"afamily":"inet","nodeEncryption":false},\{"afamily":"inet6","nodeEncryption":false}]}],"buckets":\{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":\{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":\{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":\{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":\{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":\{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":\{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":\{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":\{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":\{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":\{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":\{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":\{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":\{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":\{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":\{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":\{"percentage":30}},"tasks":\{"uri":"/pools/default/tasks?v=35395949"},"counters":\{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":\{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":\{"total":198285* Connection #0 to host 10.112.200.104 left intact 72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
            Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}
            Error when executing from postman-
            CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord

            Error when executing using curl
            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/long_chain172.16.1.174.pem)” --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information

            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/ca.pem)” --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information
            {code}

            Have verified that the certificates that were created are working.
            {code:java}curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key https://10.112.200.104:18091/pools/default
            * Trying 10.112.200.104...
            * TCP_NODELAY set
            * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
            * ALPN, offering h2
            * ALPN, offering http/1.1
            * successfully set certificate verify locations:
            * CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem
              CApath: none
            * TLSv1.2 (OUT), TLS handshake, Client hello (1):
            * TLSv1.2 (IN), TLS handshake, Server hello (2):
            * TLSv1.2 (IN), TLS handshake, Certificate (11):
            * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
            * TLSv1.2 (IN), TLS handshake, Request CERT (13):
            * TLSv1.2 (IN), TLS handshake, Server finished (14):
            * TLSv1.2 (OUT), TLS handshake, Certificate (11):
            * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
            * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
            * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
            * TLSv1.2 (OUT), TLS handshake, Finished (20):
            * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
            * TLSv1.2 (IN), TLS handshake, Finished (20):
            * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
            * ALPN, server did not agree to a protocol
            * Server certificate:
            * subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
            * start date: Jul 9 04:22:00 2020 GMT
            * expire date: Jul 9 04:22:00 2021 GMT
            * subjectAltName: host "10.112.200.104" matched cert's IP address!
            * issuer: C=UA; O=My Company; CN=My Company Intermediate CA
            * SSL certificate verify ok.
            > GET /pools/default HTTP/1.1
            > Host: 10.112.200.104:18091
            > User-Agent: curl/7.64.1
            > Accept: */*
            >
            < HTTP/1.1 200 OK
            < X-XSS-Protection: 1; mode=block
            < X-Permitted-Cross-Domain-Policies: none
            < X-Frame-Options: DENY
            < X-Content-Type-Options: nosniff
            < Server: Couchbase Server
            < Pragma: no-cache
            < Expires: Thu, 01 Jan 1970 00:00:00 GMT
            < Date: Thu, 09 Jul 2020 04:32:12 GMT
            < Content-Type: application/json
            < Content-Length: 4181
            < Cache-Control: no-cache,no-store,must-revalidate
            <
            {"name":"default","nodes":[{"systemStats":{"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]}],"buckets":{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":{"percentage":30}},"tasks":{"uri":"/pools/default/tasks?v=35395949"},"counters":{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":{"total":198285* Connection #0 to host 10.112.200.104 left intact
            72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
            {code}

            Have also verified that the above API endpoint is not open:

            {code:java}
            curl -v http://10.112.200.104:8091/pools/default
            * Trying 10.112.200.104...
            * TCP_NODELAY set
            * Connected to 10.112.200.104 (10.112.200.104) port 8091 (#0)
            > GET /pools/default HTTP/1.1
            > Host: 10.112.200.104:8091
            > User-Agent: curl/7.64.1
            > Accept: */*
            >
            < HTTP/1.1 401 Unauthorized
            < X-XSS-Protection: 1; mode=block
            < X-Permitted-Cross-Domain-Policies: none
            < X-Frame-Options: DENY
            < X-Content-Type-Options: nosniff
            < WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"
            < Server: Couchbase Server
            < Pragma: no-cache
            < Expires: Thu, 01 Jan 1970 00:00:00 GMT
            < Date: Thu, 09 Jul 2020 05:00:22 GMT
            < Content-Length: 0
            < Cache-Control: no-cache,no-store,must-revalidate
            <
            * Connection #0 to host 10.112.200.104 left intact
            * Closing connection 0
            {code}

            Attaching all the certificates that i generated.

            Node certificates -
            10.112.200.104.csr ,10.112.200.104.key , 10.112.200.104.pem

            Client certificates -
            172.16.1.174.csr, 172.16.1.174.key, 172.16.1.174.pem

            root certificates-
            ca.key, ca.pem

            Intermediate certificates-
            int.csr, int.key, int.pem, intermediateCA.srl

            other certificates-
            long_chain10.112.200.104.pem, long_chain172.16.1.174.pem, root.crt, rootCA.srl
            ritam.sharma Ritam Sharma made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            umang.agrawal Umang made changes -
            Attachment newcerts73C1.zip [ 100383 ]
            umang.agrawal Umang made changes -
            Description Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}
            Error when executing from postman-
            CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord

            Error when executing using curl
            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/long_chain172.16.1.174.pem)” --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information

            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/ca.pem)” --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information
            {code}

            Have verified that the certificates that were created are working.
            {code:java}curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key https://10.112.200.104:18091/pools/default
            * Trying 10.112.200.104...
            * TCP_NODELAY set
            * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
            * ALPN, offering h2
            * ALPN, offering http/1.1
            * successfully set certificate verify locations:
            * CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem
              CApath: none
            * TLSv1.2 (OUT), TLS handshake, Client hello (1):
            * TLSv1.2 (IN), TLS handshake, Server hello (2):
            * TLSv1.2 (IN), TLS handshake, Certificate (11):
            * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
            * TLSv1.2 (IN), TLS handshake, Request CERT (13):
            * TLSv1.2 (IN), TLS handshake, Server finished (14):
            * TLSv1.2 (OUT), TLS handshake, Certificate (11):
            * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
            * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
            * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
            * TLSv1.2 (OUT), TLS handshake, Finished (20):
            * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
            * TLSv1.2 (IN), TLS handshake, Finished (20):
            * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
            * ALPN, server did not agree to a protocol
            * Server certificate:
            * subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
            * start date: Jul 9 04:22:00 2020 GMT
            * expire date: Jul 9 04:22:00 2021 GMT
            * subjectAltName: host "10.112.200.104" matched cert's IP address!
            * issuer: C=UA; O=My Company; CN=My Company Intermediate CA
            * SSL certificate verify ok.
            > GET /pools/default HTTP/1.1
            > Host: 10.112.200.104:18091
            > User-Agent: curl/7.64.1
            > Accept: */*
            >
            < HTTP/1.1 200 OK
            < X-XSS-Protection: 1; mode=block
            < X-Permitted-Cross-Domain-Policies: none
            < X-Frame-Options: DENY
            < X-Content-Type-Options: nosniff
            < Server: Couchbase Server
            < Pragma: no-cache
            < Expires: Thu, 01 Jan 1970 00:00:00 GMT
            < Date: Thu, 09 Jul 2020 04:32:12 GMT
            < Content-Type: application/json
            < Content-Length: 4181
            < Cache-Control: no-cache,no-store,must-revalidate
            <
            {"name":"default","nodes":[{"systemStats":{"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]}],"buckets":{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":{"percentage":30}},"tasks":{"uri":"/pools/default/tasks?v=35395949"},"counters":{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":{"total":198285* Connection #0 to host 10.112.200.104 left intact
            72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
            {code}

            Have also verified that the above API endpoint is not open:

            {code:java}
            curl -v http://10.112.200.104:8091/pools/default
            * Trying 10.112.200.104...
            * TCP_NODELAY set
            * Connected to 10.112.200.104 (10.112.200.104) port 8091 (#0)
            > GET /pools/default HTTP/1.1
            > Host: 10.112.200.104:8091
            > User-Agent: curl/7.64.1
            > Accept: */*
            >
            < HTTP/1.1 401 Unauthorized
            < X-XSS-Protection: 1; mode=block
            < X-Permitted-Cross-Domain-Policies: none
            < X-Frame-Options: DENY
            < X-Content-Type-Options: nosniff
            < WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"
            < Server: Couchbase Server
            < Pragma: no-cache
            < Expires: Thu, 01 Jan 1970 00:00:00 GMT
            < Date: Thu, 09 Jul 2020 05:00:22 GMT
            < Content-Length: 0
            < Cache-Control: no-cache,no-store,must-revalidate
            <
            * Connection #0 to host 10.112.200.104 left intact
            * Closing connection 0
            {code}

            Attaching all the certificates that i generated.

            Node certificates -
            10.112.200.104.csr ,10.112.200.104.key , 10.112.200.104.pem

            Client certificates -
            172.16.1.174.csr, 172.16.1.174.key, 172.16.1.174.pem

            root certificates-
            ca.key, ca.pem

            Intermediate certificates-
            int.csr, int.key, int.pem, intermediateCA.srl

            other certificates-
            long_chain10.112.200.104.pem, long_chain172.16.1.174.pem, root.crt, rootCA.srl
            Steps to reproduce-

            1. create 2 clusters, a local cluster with cbas node, a remote cluster with KV node.

            2. generate certificates root, node and client certificates for both the cluster.

            3. create link to remote cluster with full encryption, remote cluster root cert, client cert and client key.

            4. link creation failed.
            {code:java}
            Error when executing from postman-
            CBAS0025: Link authentication failed: javax.net.ssl.SSLException: readHandshakeRecord

            Error when executing using curl
            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/long_chain172.16.1.174.pem)” --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information

            curl -v -u Administrator:password -X POST http://10.112.200.103:8095/analytics/link -d dataverse=Default -d name=myCbLink -d type=couchbase -d hostname=10.112.200.104 -d encryption=full --data-urlencode "certificate=$(cat /private/tmp/newcerts73C1/ca.pem)” --data-urlencode "clientCertificate=$(cat /private/tmp/newcerts73C1/172.16.1.174.pem)" --data-urlencode "clientKey=$(cat /private/tmp/newcerts73C1/172.16.1.174.key)”
            curl: option -----END: is unknown
            curl: try 'curl --help' or 'curl --manual' for more information
            {code}

            Have verified that the certificates that were created are working.
            {code:java}curl -v --cacert /tmp/newcerts73C1/long_chain172.16.1.174.pem --cert-type PEM --cert /tmp/newcerts73C1/172.16.1.174.pem --key-type PEM --key /tmp/newcerts73C1/172.16.1.174.key https://10.112.200.104:18091/pools/default
            * Trying 10.112.200.104...
            * TCP_NODELAY set
            * Connected to 10.112.200.104 (10.112.200.104) port 18091 (#0)
            * ALPN, offering h2
            * ALPN, offering http/1.1
            * successfully set certificate verify locations:
            * CAfile: /tmp/newcerts73C1/long_chain172.16.1.174.pem
              CApath: none
            * TLSv1.2 (OUT), TLS handshake, Client hello (1):
            * TLSv1.2 (IN), TLS handshake, Server hello (2):
            * TLSv1.2 (IN), TLS handshake, Certificate (11):
            * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
            * TLSv1.2 (IN), TLS handshake, Request CERT (13):
            * TLSv1.2 (IN), TLS handshake, Server finished (14):
            * TLSv1.2 (OUT), TLS handshake, Certificate (11):
            * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
            * TLSv1.2 (OUT), TLS handshake, CERT verify (15):
            * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
            * TLSv1.2 (OUT), TLS handshake, Finished (20):
            * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
            * TLSv1.2 (IN), TLS handshake, Finished (20):
            * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
            * ALPN, server did not agree to a protocol
            * Server certificate:
            * subject: C=UA; ST=California; L=Mountain View; O=My Company; CN=www.cbadminbucket.com
            * start date: Jul 9 04:22:00 2020 GMT
            * expire date: Jul 9 04:22:00 2021 GMT
            * subjectAltName: host "10.112.200.104" matched cert's IP address!
            * issuer: C=UA; O=My Company; CN=My Company Intermediate CA
            * SSL certificate verify ok.
            > GET /pools/default HTTP/1.1
            > Host: 10.112.200.104:18091
            > User-Agent: curl/7.64.1
            > Accept: */*
            >
            < HTTP/1.1 200 OK
            < X-XSS-Protection: 1; mode=block
            < X-Permitted-Cross-Domain-Policies: none
            < X-Frame-Options: DENY
            < X-Content-Type-Options: nosniff
            < Server: Couchbase Server
            < Pragma: no-cache
            < Expires: Thu, 01 Jan 1970 00:00:00 GMT
            < Date: Thu, 09 Jul 2020 04:32:12 GMT
            < Content-Type: application/json
            < Content-Length: 4181
            < Cache-Control: no-cache,no-store,must-revalidate
            <
            {"name":"default","nodes":[{"systemStats":{"cpu_utilization_rate":4.081632653061225,"cpu_stolen_rate":0,"swap_total":1107292160,"swap_used":6860800,"mem_total":1930829824,"mem_free":1444765696,"mem_limit":1930829824,"cpu_cores_available":1,"allocstall":3065},"interestingStats":{},"uptime":"1748","memoryTotal":1930829824,"memoryFree":1444765696,"mcdMemoryReserved":1473,"mcdMemoryAllocated":1473,"couchApiBase":"http://10.112.200.104:8092/","couchApiBaseHTTPS":"https://10.112.200.104:18092/","clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"ns_1@10.112.200.104","thisNode":true,"hostname":"10.112.200.104:8091","nodeUUID":"a11586ede8b0bb236f93edcc53006f67","clusterCompatibility":393222,"version":"6.6.0-7861-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":1,"ports":{"direct":11210,"httpsCAPI":18092,"httpsMgmt":18091,"distTCP":21100,"distTLS":21150},"services":["index","kv","n1ql"],"nodeEncryption":false,"configuredHostname":"10.112.200.104:8091","addressFamily":"inet","externalListeners":[{"afamily":"inet","nodeEncryption":false},{"afamily":"inet6","nodeEncryption":false}]}],"buckets":{"uri":"/pools/default/buckets?v=75954893&uuid=5e16dc870081e4782e515a37a644f937","terseBucketsBase":"/pools/default/b/","terseStreamingBucketsBase":"/pools/default/bs/"},"remoteClusters":{"uri":"/pools/default/remoteClusters?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/pools/default/remoteClusters?just_validate=1"},"alerts":[],"alertsSilenceURL":"/controller/resetAlerts?uuid=5e16dc870081e4782e515a37a644f937&token=0","controllers":{"addNode":{"uri":"/controller/addNodeV2?uuid=5e16dc870081e4782e515a37a644f937"},"rebalance":{"uri":"/controller/rebalance?uuid=5e16dc870081e4782e515a37a644f937"},"failOver":{"uri":"/controller/failOver?uuid=5e16dc870081e4782e515a37a644f937"},"startGracefulFailover":{"uri":"/controller/startGracefulFailover?uuid=5e16dc870081e4782e515a37a644f937"},"reAddNode":{"uri":"/controller/reAddNode?uuid=5e16dc870081e4782e515a37a644f937"},"reFailOver":{"uri":"/controller/reFailOver?uuid=5e16dc870081e4782e515a37a644f937"},"ejectNode":{"uri":"/controller/ejectNode?uuid=5e16dc870081e4782e515a37a644f937"},"setRecoveryType":{"uri":"/controller/setRecoveryType?uuid=5e16dc870081e4782e515a37a644f937"},"setAutoCompaction":{"uri":"/controller/setAutoCompaction?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/setAutoCompaction?just_validate=1"},"clusterLogsCollection":{"startURI":"/controller/startLogsCollection?uuid=5e16dc870081e4782e515a37a644f937","cancelURI":"/controller/cancelLogsCollection?uuid=5e16dc870081e4782e515a37a644f937"},"replication":{"createURI":"/controller/createReplication?uuid=5e16dc870081e4782e515a37a644f937","validateURI":"/controller/createReplication?just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=5e16dc870081e4782e515a37a644f937","nodeStatusesUri":"/nodeStatuses","maxBucketCount":30,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":{"fromHour":0,"toHour":0,"fromMinute":0,"toMinute":0,"abortOutside":false}},"indexFragmentationThreshold":{"percentage":30}},"tasks":{"uri":"/pools/default/tasks?v=35395949"},"counters":{"rebalance_success":1,"rebalance_start":1},"indexStatusURI":"/indexStatus?v=21137658","checkPermissionsURI":"/pools/default/checkPermissions?v=Ad00Y9Fmacx5sM1JEwCr8PotHjk%3D","serverGroupsUri":"/pools/default/serverGroups?v=5587421","clusterName":"","balanced":true,"memoryQuota":256,"indexMemoryQuota":256,"ftsMemoryQuota":512,"cbasMemoryQuota":1024,"eventingMemoryQuota":256,"storageTotals":{"ram":{"total":1930829824,"quotaTotal":268435456,"quotaUsed":0,"used":996306944,"usedByData":0,"quotaUsedPerNode":0,"quotaTotalPerNode":268435456},"hdd":{"total":198285* Connection #0 to host 10.112.200.104 left intact
            72160,"quotaTotal":19828572160,"used":3370857267,"usedByData":0,"free":16457714893}}}* Closing connection 0
            {code}

            Have also verified that the above API endpoint does not works without authentication:

            {code:java}
            curl -v http://10.112.200.104:8091/pools/default
            * Trying 10.112.200.104...
            * TCP_NODELAY set
            * Connected to 10.112.200.104 (10.112.200.104) port 8091 (#0)
            > GET /pools/default HTTP/1.1
            > Host: 10.112.200.104:8091
            > User-Agent: curl/7.64.1
            > Accept: */*
            >
            < HTTP/1.1 401 Unauthorized
            < X-XSS-Protection: 1; mode=block
            < X-Permitted-Cross-Domain-Policies: none
            < X-Frame-Options: DENY
            < X-Content-Type-Options: nosniff
            < WWW-Authenticate: Basic realm="Couchbase Server Admin / REST"
            < Server: Couchbase Server
            < Pragma: no-cache
            < Expires: Thu, 01 Jan 1970 00:00:00 GMT
            < Date: Thu, 09 Jul 2020 05:00:22 GMT
            < Content-Length: 0
            < Cache-Control: no-cache,no-store,must-revalidate
            <
            * Connection #0 to host 10.112.200.104 left intact
            * Closing connection 0
            {code}

            Attaching all the certificates that i generated.

            Node certificates -
            10.112.200.104.csr ,10.112.200.104.key , 10.112.200.104.pem

            Client certificates -
            172.16.1.174.csr, 172.16.1.174.key, 172.16.1.174.pem

            root certificates-
            ca.key, ca.pem

            Intermediate certificates-
            int.csr, int.key, int.pem, intermediateCA.srl

            other certificates-
            long_chain10.112.200.104.pem, long_chain172.16.1.174.pem, root.crt, rootCA.srl
            till Till Westmann made changes -
            Assignee Till Westmann [ till ] Michael Blow [ michael.blow ]
            michael.blow Michael Blow made changes -
            Assignee Michael Blow [ michael.blow ] Umang [ JIRAUSER24787 ]
            Resolution User Error [ 10100 ]
            Status Open [ 1 ] Resolved [ 5 ]
            umang.agrawal Umang made changes -
            Assignee Umang [ JIRAUSER24787 ] Michael Blow [ michael.blow ]
            umang.agrawal Umang made changes -
            Resolution User Error [ 10100 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            umang.agrawal Umang made changes -
            Attachment newcerts45C1.zip [ 100559 ]
            umang.agrawal Umang made changes -
            Link to Log File, atop/blg, CBCollectInfo, Core dump https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB40358/create_link
            till Till Westmann made changes -
            Labels analytics analytics triaged
            till Till Westmann made changes -
            Due Date 13/Jul/20
            till Till Westmann made changes -
            Sprint CX Sprint 207 [ 1145 ]
            michael.blow Michael Blow made changes -
            Status Reopened [ 4 ] In Progress [ 3 ]
            michael.blow Michael Blow made changes -
            Resolution User Error [ 10100 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            umang.agrawal Umang made changes -
            Assignee Michael Blow [ michael.blow ] Umang [ JIRAUSER24787 ]
            Status Resolved [ 5 ] Closed [ 6 ]
            michael.blow Michael Blow made changes -
            Labels analytics triaged analytics test-change-only triaged
            till Till Westmann made changes -
            Link This issue relates to MB-40347 [ MB-40347 ]

              People

              Assignee:
              umang.agrawal Umang
              Reporter:
              umang.agrawal Umang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Due:
                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty