Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42637

Secure (https) Analytics S3 Remote Links are broken when node-to-node encryption is enabled

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      No
    • Sprint:
      CX Sprint 225

      Description

      Summary
      When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

      Steps to Reproduce

      • Create cluster with a single Analytics + Data node
      • Disable automatic failover:

        /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0
        

      • Enable node-to-node encryption:

        /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable
        

      • Set encryption level to 'all' (unsure if this is required):

        /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all
        

      • Create an external s3 link (you don't need actual working creds, you can just copy paste this):

        curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2
        

      • Try to create an external dataset using this s3 link:

        CREATE EXTERNAL DATASET S3productreviews
        ON `cbc-remote-links-test`
        AT s3Link
        USING "reviews"
        WITH { "format": "json", "include": "*.json" } ;
        

      Expected Behavior
      Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

      Actual Behavior
      The query returns the error:

      [
        {
          "code": 24086,
          "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
          "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"
        }
      ]
      


      This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.

      Workaround
      Do not use HTTPS for the s3 connection by overriding the serviceEndpoint when creating the link to be http://s3.amazonaws.com, do note that all data retrieved from S3 will go over the network unencrypted.

      e.g.:

      curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2 -d serviceEndpoint=http://s3.amazonaws.com
      

        Attachments

          Issue Links

          For Gerrit Dashboard: MB-42637
          # Subject Branch Project Status CR V

            Activity

            Hide
            Hussain.Towaileb Hussain Towaileb added a comment - - edited

            I managed to reproduce this locally, the below step is required to reproduce it:

            ./couchbase-cli setting-security -c localhost -u Administrator -p password --set --cluster-encryption-level all
            

            Show
            Hussain.Towaileb Hussain Towaileb added a comment - - edited I managed to reproduce this locally, the below step is required to reproduce it: ./couchbase-cli setting-security -c localhost -u Administrator -p password --set --cluster-encryption-level all
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9184 contains cbas-core commit 565a67f with commit message:
            MB-42637: Adapt tests to new serializable classes

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9184 contains cbas-core commit 565a67f with commit message: MB-42637 : Adapt tests to new serializable classes
            Hide
            murtadha.hubail Murtadha Hubail added a comment -

            Hi Matt Carabine,

            This should be fixed starting from buildĀ 6.6.1-9184.

            Hi Umang,

            We need to add an automated test for external/remote links (over HTTPs) when node-to-node encryption is enabled. In addition, we need to test all upgrades scenarios (offline/online + mixed mode) with node-2-node encryption enabled from 6.6.0 to 6.6.1 to ensure that this fix doesn't break backward compatibility in mixed mode.

            Show
            murtadha.hubail Murtadha Hubail added a comment - Hi Matt Carabine , This should be fixed starting from buildĀ 6.6.1-9184. Hi Umang , We need to add an automated test for external/remote links (over HTTPs) when node-to-node encryption is enabled. In addition, we need to test all upgrades scenarios (offline/online + mixed mode) with node-2-node encryption enabled from 6.6.0 to 6.6.1 to ensure that this fix doesn't break backward compatibility in mixed mode.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3774 contains cbas-core commit 565a67f with commit message:
            MB-42637: Adapt tests to new serializable classes

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3774 contains cbas-core commit 565a67f with commit message: MB-42637 : Adapt tests to new serializable classes
            Hide
            umang.agrawal Umang added a comment -

            Verified with couchbase server build 6.6.1-9194

            Show
            umang.agrawal Umang added a comment - Verified with couchbase server build 6.6.1-9194
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit 565a67f with commit message:
            MB-42637: Adapt tests to new serializable classes

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 565a67f with commit message: MB-42637 : Adapt tests to new serializable classes

              People

              Assignee:
              umang.agrawal Umang
              Reporter:
              matt.carabine Matt Carabine
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty