Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42277

Internal error is raised when connecting remote link created with full encryption

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      Unknown
    • Sprint:
      CX Sprint 222, CX Sprint 223

      Description

      Steps to reproduce -
      1. Setup 2 clusters, 1 analytics cluster and 1 remote data cluster.
      2. Setup certificates on remote data cluster.
      3. Load travel-sample bucket on remote data cluster.
      4. Create remote link on analytics cluster to remote cluster with encryption set to full. (create a link using both (username, password and root cert) and (root cert, client cert and client key))
      5. Connect the links created above.
      6. Internal error is raised.

      In newcerts88C2.zip -
      root cert - > root.crt
      client cert -> long_chain172.16.1.174.pem
      client key -> 172.16.1.174.key

      2020-10-26T11:34:42.484+00:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-13] handleException: unexpected exception CBAS0029: Connect link failed {"Default.r5.travel-sample" : "Internal error"}: <ud>{"host":"10.112.205.102:8091","path":"/query/service","statement":"connect link r5;","pretty":false,"mode":"immediate","clientContextID":"f7939918-9732-4347-97d0-df13a35c7c05","format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":true,"job":false,"profile":"counts","signature":true,"multiStatement":false,"parseOnly":false,"readOnly":false,"maxWarnings":10,"scanConsistency":"not_bounded","scanWait":null}</ud>
      com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.r5.travel-sample" : "Internal error"}
              at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:371) ~[cbas-connector.jar:6.6.1-9143]
              at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:65) ~[cbas-connector.jar:6.6.1-9143]
              at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:430) ~[asterix-app.jar:6.6.1-9143]
              at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:155) ~[asterix-app.jar:6.6.1-9143]
              at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app.jar:6.6.1-9143]
              at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:68) ~[hyracks-control-cc.jar:6.6.1-9143]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
              at java.lang.Thread.run(Unknown Source) [?:?]
      
      

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            umang.agrawal Umang created issue -
            umang.agrawal Umang made changes -
            Field Original Value New Value
            Description Steps to reproduce -
            1. Setup 2 clusters, 1 analytics cluster and 1 remote data cluster.
            2. Setup certificates on remote data cluster.
            3. Load travel-sample bucket on remote data cluster.
            4. Create remote link on analytics cluster to remote cluster with encryption set to full. (create a link using both (username, password and root cert) and (root cert, client cert and client key))
            5. Connect the links created above.
            6. Internal error is raised.

            In newcerts88C2.zip -
            Steps to reproduce -
            1. Setup 2 clusters, 1 analytics cluster and 1 remote data cluster.
            2. Setup certificates on remote data cluster.
            3. Load travel-sample bucket on remote data cluster.
            4. Create remote link on analytics cluster to remote cluster with encryption set to full. (create a link using both (username, password and root cert) and (root cert, client cert and client key))
            5. Connect the links created above.
            6. Internal error is raised.

            In newcerts88C2.zip -
            root cert - > root.crt
            client cert -> long_chain172.16.1.174.pem
            client key -> 172.16.1.174.key
            umang.agrawal Umang made changes -
            Attachment internal_err.zip [ 113020 ]
            umang.agrawal Umang made changes -
            Description Steps to reproduce -
            1. Setup 2 clusters, 1 analytics cluster and 1 remote data cluster.
            2. Setup certificates on remote data cluster.
            3. Load travel-sample bucket on remote data cluster.
            4. Create remote link on analytics cluster to remote cluster with encryption set to full. (create a link using both (username, password and root cert) and (root cert, client cert and client key))
            5. Connect the links created above.
            6. Internal error is raised.

            In newcerts88C2.zip -
            root cert - > root.crt
            client cert -> long_chain172.16.1.174.pem
            client key -> 172.16.1.174.key
            Steps to reproduce -
            1. Setup 2 clusters, 1 analytics cluster and 1 remote data cluster.
            2. Setup certificates on remote data cluster.
            3. Load travel-sample bucket on remote data cluster.
            4. Create remote link on analytics cluster to remote cluster with encryption set to full. (create a link using both (username, password and root cert) and (root cert, client cert and client key))
            5. Connect the links created above.
            6. Internal error is raised.

            In newcerts88C2.zip -
            root cert - > root.crt
            client cert -> long_chain172.16.1.174.pem
            client key -> 172.16.1.174.key


            {code:java}
            2020-10-26T11:34:42.484+00:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-13] handleException: unexpected exception CBAS0029: Connect link failed {"Default.r5.travel-sample" : "Internal error"}: <ud>{"host":"10.112.205.102:8091","path":"/query/service","statement":"connect link r5;","pretty":false,"mode":"immediate","clientContextID":"f7939918-9732-4347-97d0-df13a35c7c05","format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":true,"job":false,"profile":"counts","signature":true,"multiStatement":false,"parseOnly":false,"readOnly":false,"maxWarnings":10,"scanConsistency":"not_bounded","scanWait":null}</ud>
            com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.r5.travel-sample" : "Internal error"}
                    at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:371) ~[cbas-connector.jar:6.6.1-9143]
                    at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:65) ~[cbas-connector.jar:6.6.1-9143]
                    at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:430) ~[asterix-app.jar:6.6.1-9143]
                    at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:155) ~[asterix-app.jar:6.6.1-9143]
                    at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app.jar:6.6.1-9143]
                    at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:68) ~[hyracks-control-cc.jar:6.6.1-9143]
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
                    at java.lang.Thread.run(Unknown Source) [?:?]

            {code}
            till Till Westmann made changes -
            Labels analytics analytics triaged
            Hide
            till Till Westmann added a comment - - edited

            Umang, was the link created from the UI or directly using the REST API?
            Also, is this a regression?

            Show
            till Till Westmann added a comment - - edited Umang , was the link created from the UI or directly using the REST API? Also, is this a regression?
            till Till Westmann made changes -
            Link This issue relates to MB-42037 [ MB-42037 ]
            till Till Westmann made changes -
            Assignee Till Westmann [ till ] Hussain Towaileb [ hussain.towaileb ]
            Hide
            michael.blow Michael Blow added a comment -

            root cause:

            Caused by: sun.security.validator.ValidatorException: No trusted certificate found
            	at sun.security.validator.SimpleValidator.buildTrustedChain(Unknown Source) ~[?:?]
            	at sun.security.validator.SimpleValidator.engineValidate(Unknown Source) ~[?:?]
            	at sun.security.validator.Validator.validate(Unknown Source) ~[?:?]
            	at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source) ~[?:?]
            	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source) ~[?:?]
            	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source) ~[?:?]
            	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source) ~[?:?]
            	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source) ~[?:?]
            	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source) ~[?:?]
            	at sun.security.ssl.SSLHandshake.consume(Unknown Source) ~[?:?]
            	at sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:?]
            	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?]
            	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?]
            	at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
            	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source) ~[?:?]
            	at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1510) ~[core-io-1.7.14.jar:?]
            	at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1524) ~[core-io-1.7.14.jar:?]
            	at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1408) ~[core-io-1.7.14.jar:?]
            	at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[core-io-1.7.14.jar:?]
            	at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) ~[core-io-1.7.14.jar:?]
            
            

            Show
            michael.blow Michael Blow added a comment - root cause: Caused by: sun.security.validator.ValidatorException: No trusted certificate found at sun.security.validator.SimpleValidator.buildTrustedChain(Unknown Source) ~[?:?] at sun.security.validator.SimpleValidator.engineValidate(Unknown Source) ~[?:?] at sun.security.validator.Validator.validate(Unknown Source) ~[?:?] at sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source) ~[?:?] at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source) ~[?:?] at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source) ~[?:?] at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(Unknown Source) ~[?:?] at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(Unknown Source) ~[?:?] at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(Unknown Source) ~[?:?] at sun.security.ssl.SSLHandshake.consume(Unknown Source) ~[?:?] at sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:?] at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?] at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source) ~[?:?] at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1510) ~[core-io-1.7.14.jar:?] at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1524) ~[core-io-1.7.14.jar:?] at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1408) ~[core-io-1.7.14.jar:?] at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[core-io-1.7.14.jar:?] at com.couchbase.client.deps.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) ~[core-io-1.7.14.jar:?]
            Hussain.Towaileb Hussain Towaileb made changes -
            Sprint CX Sprint 222 [ 1288 ]
            Hussain.Towaileb Hussain Towaileb made changes -
            Rank Ranked higher
            Hussain.Towaileb Hussain Towaileb made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hussain.Towaileb Hussain Towaileb made changes -
            Rank Ranked higher
            Hide
            umang.agrawal Umang added a comment -

            Till Westmann It is happening for both links created from UI as well as REST API.
            Found this issue while verifying Jira issue related to creation of links with full encryption from UI.

            Show
            umang.agrawal Umang added a comment - Till Westmann It is happening for both links created from UI as well as REST API. Found this issue while verifying Jira issue related to creation of links with full encryption from UI.
            Hide
            michael.blow Michael Blow added a comment - - edited

            Umang,
            >> Is this a Regression?:Unknown

            1. Is this a regression?
            2. If so, on which build did this test last work?
            3. Was this test run at 6.6.0?

            Show
            michael.blow Michael Blow added a comment - - edited Umang , >> Is this a Regression?:Unknown 1. Is this a regression? 2. If so, on which build did this test last work? 3. Was this test run at 6.6.0?
            Hide
            till Till Westmann added a comment -

            Umang, a few more questions:

            • In step 4. we create 2 links for (username, password, root cert) and (root cert, client cert, client key). Do both fail or just the 2nd one?
            • Are there other certificate-based tests that pass?
            Show
            till Till Westmann added a comment - Umang , a few more questions: In step 4. we create 2 links for (username, password, root cert) and (root cert, client cert, client key). Do both fail or just the 2nd one? Are there other certificate-based tests that pass?
            till Till Westmann made changes -
            Sprint CX Sprint 222 [ 1288 ] CX Sprint 222, CX Sprint 223 [ 1288, 1291 ]
            Hide
            umang.agrawal Umang added a comment -

            Michael Blow
            1. Is this a regression?
            I found this issue while verifying another issue, i think we can call it regression.
            2. If so, on which build did this test last work?
            6.6.0-7904
            3. Was this test run at 6.6.0?
            Yes

            Till Westmann
            1. In step 4. we create 2 links for (username, password, root cert) and (root cert, client cert, client key). Do both fail or just the 2nd one?
            Yes they both failed.

            2. Are there other certificate-based tests that pass?
            Ritam ran a basic test yesterday to verify certificates are working or not, you can find results here http://qa.sc.couchbase.com/job/test_suite_executor/268920/console. Though i tried XDCR with certs, but there also I was able to create a link to remote cluster, but replication was failing.

            Show
            umang.agrawal Umang added a comment - Michael Blow 1. Is this a regression? I found this issue while verifying another issue, i think we can call it regression. 2. If so, on which build did this test last work? 6.6.0-7904 3. Was this test run at 6.6.0? Yes Till Westmann 1. In step 4. we create 2 links for (username, password, root cert) and (root cert, client cert, client key). Do both fail or just the 2nd one? Yes they both failed. 2. Are there other certificate-based tests that pass? Ritam ran a basic test yesterday to verify certificates are working or not, you can find results here http://qa.sc.couchbase.com/job/test_suite_executor/268920/console . Though i tried XDCR with certs, but there also I was able to create a link to remote cluster, but replication was failing.
            Hide
            umang.agrawal Umang added a comment -

            Till WestmannMichael Blow
            I am able to create and connect link with full encryption now. There seems to be some issue with my local setup, so I tried on machines on which we run our automation suites and it worked there.
            I guess we can close out this issue.
            Verfied with build 6.6.1-9155.

            Show
            umang.agrawal Umang added a comment - Till Westmann Michael Blow I am able to create and connect link with full encryption now. There seems to be some issue with my local setup, so I tried on machines on which we run our automation suites and it worked there. I guess we can close out this issue. Verfied with build 6.6.1-9155.
            till Till Westmann made changes -
            Link This issue blocks MB-40528 [ MB-40528 ]
            till Till Westmann made changes -
            Labels analytics triaged analytics approved-for-6.6.1 triaged
            Hide
            Hussain.Towaileb Hussain Towaileb added a comment - - edited

            The issue was due to a script not reloading the certificates on all servers.

            The linked fix in this issue is to report a proper message instead of "internal error" if this failure happens.

            Show
            Hussain.Towaileb Hussain Towaileb added a comment - - edited The issue was due to a script not reloading the certificates on all servers. The linked fix in this issue is to report a proper message instead of "internal error" if this failure happens.
            Hussain.Towaileb Hussain Towaileb made changes -
            Resolution Not a Bug [ 10200 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9160 contains cbas-core commit 1ad3383 with commit message:
            MB-42277: Properly handle remote link certificate failures

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9160 contains cbas-core commit 1ad3383 with commit message: MB-42277 : Properly handle remote link certificate failures
            Hide
            umang.agrawal Umang added a comment -

            This issue was happening because of not reloading the node certs in automation scripts. After reloading the certs the link connection is happening successfully.

            Show
            umang.agrawal Umang added a comment - This issue was happening because of not reloading the node certs in automation scripts. After reloading the certs the link connection is happening successfully.
            umang.agrawal Umang made changes -
            Assignee Hussain Towaileb [ hussain.towaileb ] Umang [ JIRAUSER24787 ]
            Status Resolved [ 5 ] Closed [ 6 ]
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3636 contains cbas-core commit 1ad3383 with commit message:
            MB-42277: Properly handle remote link certificate failures

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3636 contains cbas-core commit 1ad3383 with commit message: MB-42277 : Properly handle remote link certificate failures
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit 1ad3383 with commit message:
            MB-42277: Properly handle remote link certificate failures

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 1ad3383 with commit message: MB-42277 : Properly handle remote link certificate failures

              People

              Assignee:
              umang.agrawal Umang
              Reporter:
              umang.agrawal Umang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty