Secure (https) Analytics S3 Remote Links are broken when node-to-node encryption is enabled

Summary
When node-to-node encryption is enabled within the Couchbase Server cluster, all queries using a secure service endpoint (the default) to an s3 link fail.

Steps to Reproduce

• Create cluster with a single Analytics + Data node
• Disable automatic failover:

  /opt/couchbase/bin/couchbase-cli setting-autofailover -c localhost -u Administrator -p password --enable-auto-failover 0

• Enable node-to-node encryption:

  /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c localhost -u Administrator -p password --enable

• Set encryption level to 'all' (unsure if this is required):

  /opt/couchbase/bin/couchbase-cli -c localhost -u Administrator -p password --cluster-encryption-level all

• Create an external s3 link (you don't need actual working creds, you can just copy paste this):

  curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2

• Try to create an external dataset using this s3 link:

  CREATE EXTERNAL DATASET S3productreviews

  ON `cbc-remote-links-test`

  AT s3Link

  USING "reviews"

  WITH { "format": "json", "include": "*.json" } ;

Expected Behavior
Dataset is created (assuming correct keys etc), at least an error about AWS auth if not quite correct.

Actual Behavior
The query returns the error:

[

  {

    "code": 24086,

    "msg": "External source error. Unable to execute HTTP request: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",

    "query_from_user": "CREATE EXTERNAL DATASET S3productreviews5\nON `cbc-remote-links-test`\nAT mattS3LinkEnd\nUSING \"reviews\"\nWITH { \"format\": \"json\", \"include\": \"*.json\" } ;"

  }

]

This is because it cannot validate the certificate returned by the S3 endpoint due to trust store issues, likely as a result of some changes applied to trusting the server root CA for node to node encryption.

Workaround
Do not use HTTPS for the s3 connection by overriding the serviceEndpoint when creating the link to be http://s3.amazonaws.com, do note that all data retrieved from S3 will go over the network unencrypted.

e.g.:

curl -u Administrator:password -X POST "http://localhost:8095/analytics/link" -d dataverse=Default -d name=s3Link -d type=S3 -d accessKeyId=abcd --data-urlencode secretAccessKey=abcd-d region=us-east-2 -d serviceEndpoint=http://s3.amazonaws.com