Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42637

Secure (https) Analytics S3 Remote Links are broken when node-to-node encryption is enabled

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      No
    • Sprint:
      CX Sprint 225

      Description

      Summary
      When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

      Steps to Reproduce

      • Create cluster with a single Analytics + Data node
      • Disable automatic failover:

        /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0
        

      • Enable node-to-node encryption:

        /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable
        

      • Set encryption level to 'all' (unsure if this is required):

        /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all
        

      • Create an external s3 link (you don't need actual working creds, you can just copy paste this):

        curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2
        

      • Try to create an external dataset using this s3 link:

        CREATE EXTERNAL DATASET S3productreviews
        ON `cbc-remote-links-test`
        AT s3Link
        USING "reviews"
        WITH { "format": "json", "include": "*.json" } ;
        

      Expected Behavior
      Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

      Actual Behavior
      The query returns the error:

      [
        {
          "code": 24086,
          "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
          "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"
        }
      ]
      


      This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.

      Workaround
      Do not use HTTPS for the s3 connection by overriding the serviceEndpoint when creating the link to be http://s3.amazonaws.com, do note that all data retrieved from S3 will go over the network unencrypted.

      e.g.:

      curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2 -d serviceEndpoint=http://s3.amazonaws.com
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            matt.carabine Matt Carabine created issue -
            matt.carabine Matt Carabine made changes -
            Field Original Value New Value
            Description +Summary+
            When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

            +Steps to Reproduce+
            - Create cluster with a single Analytics + Data node
            - Disable automatic failover: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0
            {noformat}
            - Enable node-to-node encryption: {noformat}
            /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable
            {noformat}
            - Set encryption level to 'all' (unsure if this is required): {noformat}
            /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all
            {noformat}
            - Create an external s3 link (you don't need actual working creds, you can just copy paste this): {noformat}
            curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2
            {noformat}
            - Try to create an external dataset using this s3 link: {noformat}
            CREATE EXTERNAL DATASET S3productreviews
            ON `cbc-remote-links-test`
            AT s3Link
            USING "reviews"
            WITH { "format": "json", "include": "*.json" } ;
            {noformat}

            +Expected Behavior+
            Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

            +Actual Behavior+
            The query returns the error: {noformat}
            [
              {
                "code": 24086,
                "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
                "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"
              }
            ]
            {noformat}
            This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.
            +Summary+
            When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

            +Steps to Reproduce+
            - Create cluster with a single Analytics + Data node
            - Disable automatic failover: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0
            {noformat}
            - Enable node-to-node encryption: {noformat}
            /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable
            {noformat}
            - Set encryption level to 'all' (unsure if this is required): {noformat}
            /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all
            {noformat}
            - Create an external s3 link (you don't need actual working creds, you can just copy paste this): {noformat}
            curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2
            {noformat}
            - Try to create an external dataset using this s3 link: {noformat}
            CREATE EXTERNAL DATASET S3productreviews
            ON `cbc-remote-links-test`
            AT s3Link
            USING "reviews"
            WITH { "format": "json", "include": "*.json" } ;
            {noformat}

            +Expected Behavior+
            Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

            +Actual Behavior+
            The query returns the error: {noformat}
            [
              {
                "code": 24086,
                "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
                "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"
              }
            ]
            {noformat}
            This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.

            +Workaround+
            Do not use HTTPS for the s3 connection by overriding the serviceEndpoint when creating the link to be {{http://s3.amazonaws.com}}, do note that all data retrieved from S3 will go over the network unencrypted.
            matt.carabine Matt Carabine made changes -
            Description +Summary+
            When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

            +Steps to Reproduce+
            - Create cluster with a single Analytics + Data node
            - Disable automatic failover: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0
            {noformat}
            - Enable node-to-node encryption: {noformat}
            /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable
            {noformat}
            - Set encryption level to 'all' (unsure if this is required): {noformat}
            /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all
            {noformat}
            - Create an external s3 link (you don't need actual working creds, you can just copy paste this): {noformat}
            curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2
            {noformat}
            - Try to create an external dataset using this s3 link: {noformat}
            CREATE EXTERNAL DATASET S3productreviews
            ON `cbc-remote-links-test`
            AT s3Link
            USING "reviews"
            WITH { "format": "json", "include": "*.json" } ;
            {noformat}

            +Expected Behavior+
            Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

            +Actual Behavior+
            The query returns the error: {noformat}
            [
              {
                "code": 24086,
                "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
                "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"
              }
            ]
            {noformat}
            This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.

            +Workaround+
            Do not use HTTPS for the s3 connection by overriding the serviceEndpoint when creating the link to be {{http://s3.amazonaws.com}}, do note that all data retrieved from S3 will go over the network unencrypted.
            +Summary+
            When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

            +Steps to Reproduce+
            - Create cluster with a single Analytics + Data node
            - Disable automatic failover: {noformat}
            /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0
            {noformat}
            - Enable node-to-node encryption: {noformat}
            /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable
            {noformat}
            - Set encryption level to 'all' (unsure if this is required): {noformat}
            /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all
            {noformat}
            - Create an external s3 link (you don't need actual working creds, you can just copy paste this): {noformat}
            curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2
            {noformat}
            - Try to create an external dataset using this s3 link: {noformat}
            CREATE EXTERNAL DATASET S3productreviews
            ON `cbc-remote-links-test`
            AT s3Link
            USING "reviews"
            WITH { "format": "json", "include": "*.json" } ;
            {noformat}

            +Expected Behavior+
            Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

            +Actual Behavior+
            The query returns the error: {noformat}
            [
              {
                "code": 24086,
                "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
                "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"
              }
            ]
            {noformat}
            This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.

            +Workaround+
            Do not use HTTPS for the s3 connection by overriding the serviceEndpoint when creating the link to be {{http://s3.amazonaws.com}}, do note that all data retrieved from S3 will go over the network unencrypted.

            e.g.: {noformat}
            curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2 -d serviceEndpoint=http://s3.amazonaws.com
            {noformat}
            matt.carabine Matt Carabine made changes -
            Link This issue blocks CBSE-9193 [ CBSE-9193 ]
            ianmccloy Ian McCloy made changes -
            Labels security
            till Till Westmann made changes -
            Assignee Till Westmann [ till ] Hussain Towaileb [ hussain.towaileb ]
            till Till Westmann made changes -
            Fix Version/s 6.6.1 [ 17002 ]
            till Till Westmann made changes -
            Labels security security triaged
            till Till Westmann made changes -
            Rank Ranked higher
            till Till Westmann made changes -
            Rank Ranked higher
            till Till Westmann made changes -
            Sprint CX Sprint 225 [ 1314 ]
            till Till Westmann made changes -
            Rank Ranked lower
            Hussain.Towaileb Hussain Towaileb made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            till Till Westmann made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-40528 [ MB-40528 ]
            wayne Wayne Siu made changes -
            Labels security triaged approved-for-6.6.1 security triaged
            murtadha.hubail Murtadha Hubail made changes -
            Remote Link This issue links to "AsterixDB commit (Web Link)" [ 21106 ]
            Hussain.Towaileb Hussain Towaileb made changes -
            Assignee Hussain Towaileb [ hussain.towaileb ] Murtadha Hubail [ murtadha.hubail ]
            murtadha.hubail Murtadha Hubail made changes -
            Assignee Murtadha Hubail [ murtadha.hubail ] Umang [ JIRAUSER24787 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            umang.agrawal Umang made changes -
            VERIFICATION STEPS Ran automation suites with node-to-node encryption enabled.
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              Assignee:
              umang.agrawal Umang
              Reporter:
              matt.carabine Matt Carabine
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty