Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42037

Internal error is raised while reconnecting remote link if remote server is shutdown

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      No
    • Sprint:
      CX Sprint 221, CX Sprint 222, CX Sprint 223

      Description

      Steps to reproduce-

      1) Load travel-sample bucket in remote cluster.

      2) Create remote link from webUI.

      3) Create dataset on `travel-sample` bucket in remote cluster.

      4) Connect remote link.

      5) Bring down the remote server.

      6) Disconnect remote link.

      7) Try reconnecting remote link.

      Following error is raised - 

      [
        {
          "code": 22001,
          "msg": "Connect link failed {\"Default.abc.travel-sample\" : \"Internal error\"}",
          "query_from_user": "connect link abc;"
        }
      ]
      

      2020-10-17T12:33:28.479+00:00 WARN CBAS.apache.asterix [Executor-6:ClusterController] CBAS0029: Connect link failed {"Default.abc.travel-sample" : "Internal error"}
      com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.abc.travel-sample" : "Internal error"}
              at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:371) ~[cbas-connector.jar:6.6.1-9124]
              at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:65) ~[cbas-connector.jar:6.6.1-9124]
              at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:430) ~[asterix-app.jar:6.6.1-9124]
              at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:155) ~[asterix-app.jar:6.6.1-9124]
              at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app.jar:6.6.1-9124]
              at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:68) ~[hyracks-control-cc.jar:6.6.1-9124]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
              at java.lang.Thread.run(Unknown Source) [?:?]
      2020-10-17T12:33:28.482+00:00 INFO CBAS.messaging.NCMessageBroker [Worker:c69e9d555004ff3faec46e4bb11b1d44] Received message: ExecuteStatementResponseMessage(id=134): 0 characters
      2020-10-17T12:33:28.482+00:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-0] handleException: unexpected exception CBAS0029: Connect link failed {"Default.abc.travel-sample" : "Internal error"}: <ud>{"host":"10.112.205.102:8091","path":"/query/service","statement":"connect link abc;","pretty":false,"mode":"immediate","clientContextID":"54df2870-30c9-4665-9aae-e55895f64376","format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":true,"job":false,"profile":"counts","signature":true,"multiStatement":false,"parseOnly":false,"readOnly":false,"maxWarnings":10,"scanConsistency":"not_bounded","scanWait":null}</ud>
      com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.abc.travel-sample" : "Internal error"}
              at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:371) ~[cbas-connector.jar:6.6.1-9124]
              at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:65) ~[cbas-connector.jar:6.6.1-9124]
              at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:430) ~[asterix-app.jar:6.6.1-9124]
              at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:155) ~[asterix-app.jar:6.6.1-9124]
              at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app.jar:6.6.1-9124]
              at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:68) ~[hyracks-control-cc.jar:6.6.1-9124]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
              at java.lang.Thread.run(Unknown Source) [?:?]
      

        Attachments

        1. internal_err.zip
          10.18 MB
        2. internal_err.zip
          83.64 MB

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9141 contains cbas-core commit de25d8f with commit message:
            MB-42037: Properly handle failure when remote server is down

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9141 contains cbas-core commit de25d8f with commit message: MB-42037 : Properly handle failure when remote server is down
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9143 contains cbas-core commit 99142ff with commit message:
            MB-42037: Properly display error messages from remote clusters

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9143 contains cbas-core commit 99142ff with commit message: MB-42037 : Properly display error messages from remote clusters
            Hide
            umang.agrawal Umang added a comment -

            This issue is still seen in couchbase server build 6.6.1-9143

             

            ==> analytics_info.log <==
            2020-10-25T10:40:32.307+00:00 INFO CBAS.messaging.NCMessageBroker [Worker:b3b0f2cd0d302b74976ee4c8e346df26] Received message: ExecuteStatementResponseMessage(id=139): 0 characters
            2020-10-25T10:40:32.307+00:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-7] handleException: unexpected exception CBAS0029: Connect link failed {"Default.l1.beer-sample" : "Internal error"}: <ud>{"host":"10.112.205.102:8091","path":"/query/service","statement":"connect link l1;","pretty":false,"mode":"immediate","clientContextID":"9a166005-c574-428f-a003-db5a76878c6e","format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":true,"job":false,"profile":"counts","signature":true,"multiStatement":false,"parseOnly":false,"readOnly":false,"maxWarnings":10,"scanConsistency":"not_bounded","scanWait":null}</ud>
            com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed {"Default.l1.beer-sample" : "Internal error"}
            	at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java:371) ~[cbas-connector.jar:6.6.1-9143]
            	at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java:65) ~[cbas-connector.jar:6.6.1-9143]
            	at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java:430) ~[asterix-app.jar:6.6.1-9143]
            	at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java:155) ~[asterix-app.jar:6.6.1-9143]
            	at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java:64) ~[asterix-app.jar:6.6.1-9143]
            	at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$0(ApplicationMessageWork.java:68) ~[hyracks-control-cc.jar:6.6.1-9143]
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
            	at java.lang.Thread.run(Unknown Source) [?:?]
            

            Show
            umang.agrawal Umang added a comment - This issue is still seen in couchbase server build 6.6.1-9143   ==> analytics_info.log <== 2020 - 10 -25T10: 40 : 32.307 + 00 : 00 INFO CBAS.messaging.NCMessageBroker [Worker:b3b0f2cd0d302b74976ee4c8e346df26] Received message: ExecuteStatementResponseMessage(id= 139 ): 0 characters 2020 - 10 -25T10: 40 : 32.307 + 00 : 00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port: 8095 )- 7 ] handleException: unexpected exception CBAS0029: Connect link failed { "Default.l1.beer-sample" : "Internal error" }: <ud>{ "host" : "10.112.205.102:8091" , "path" : "/query/service" , "statement" : "connect link l1;" , "pretty" : false , "mode" : "immediate" , "clientContextID" : "9a166005-c574-428f-a003-db5a76878c6e" , "format" : "CLEAN_JSON" , "timeout" : 9223372036854775807 , "maxResultReads" : 1 , "planFormat" : "JSON" , "expressionTree" : false , "rewrittenExpressionTree" : false , "logicalPlan" : false , "optimizedLogicalPlan" : true , "job" : false , "profile" : "counts" , "signature" : true , "multiStatement" : false , "parseOnly" : false , "readOnly" : false , "maxWarnings" : 10 , "scanConsistency" : "not_bounded" , "scanWait" : null }</ud> com.couchbase.analytics.common.exceptions.AnalyticsHyracksException: CBAS0029: Connect link failed { "Default.l1.beer-sample" : "Internal error" } at com.couchbase.analytics.lang.ConnectLinkStatement.doHandle(ConnectLinkStatement.java: 371 ) ~[cbas-connector.jar: 6.6 . 1 - 9143 ] at com.couchbase.analytics.lang.ConnectionStatement.handle(ConnectionStatement.java: 65 ) ~[cbas-connector.jar: 6.6 . 1 - 9143 ] at org.apache.asterix.app.translator.QueryTranslator.compileAndExecute(QueryTranslator.java: 430 ) ~[asterix-app.jar: 6.6 . 1 - 9143 ] at org.apache.asterix.app.message.ExecuteStatementRequestMessage.handle(ExecuteStatementRequestMessage.java: 155 ) ~[asterix-app.jar: 6.6 . 1 - 9143 ] at org.apache.asterix.messaging.CCMessageBroker.receivedMessage(CCMessageBroker.java: 64 ) ~[asterix-app.jar: 6.6 . 1 - 9143 ] at org.apache.hyracks.control.cc.work.ApplicationMessageWork.lambda$notifyMessageBroker$ 0 (ApplicationMessageWork.java: 68 ) ~[hyracks-control-cc.jar: 6.6 . 1 - 9143 ] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?]
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9145 contains cbas-core commit ad5211b with commit message:
            MB-42037: Rethrow error if it is already handled properly

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9145 contains cbas-core commit ad5211b with commit message: MB-42037 : Rethrow error if it is already handled properly
            Hide
            michael.blow Michael Blow added a comment -

            a previous fix caused a regression on behavior on bucket UUID change

            Show
            michael.blow Michael Blow added a comment - a previous fix caused a regression on behavior on bucket UUID change
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.1-9163 contains cbas-core commit 8c33722 with commit message:
            MB-42037: don't encapsulate product exceptions

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.1-9163 contains cbas-core commit 8c33722 with commit message: MB-42037 : don't encapsulate product exceptions
            Hide
            umang.agrawal Umang added a comment -

            Verified with build 6.6.1-9164.
            Following error is now being displayed-

            [
              {
                "code": 22001,
                "msg": "Connect link failed {\"Default.q1.travel-sample\" : \"Cannot connect to host for link Default.q1: java.net.ConnectException: Connection refused (Connection refused)\"}",
                "query_from_user": "connect link q1;"
              }
            ]
            

            Show
            umang.agrawal Umang added a comment - Verified with build 6.6.1-9164. Following error is now being displayed- [ { "code" : 22001 , "msg" : "Connect link failed {\"Default.q1.travel-sample\" : \"Cannot connect to host for link Default.q1: java.net.ConnectException: Connection refused (Connection refused)\"}" , "query_from_user" : "connect link q1;" } ]
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3636 contains cbas-core commit 8c33722 with commit message:
            MB-42037: don't encapsulate product exceptions

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3636 contains cbas-core commit 8c33722 with commit message: MB-42037 : don't encapsulate product exceptions
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3636 contains cbas-core commit ad5211b with commit message:
            MB-42037: Rethrow error if it is already handled properly

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3636 contains cbas-core commit ad5211b with commit message: MB-42037 : Rethrow error if it is already handled properly
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3636 contains cbas-core commit 99142ff with commit message:
            MB-42037: Properly display error messages from remote clusters

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3636 contains cbas-core commit 99142ff with commit message: MB-42037 : Properly display error messages from remote clusters
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-3636 contains cbas-core commit de25d8f with commit message:
            MB-42037: Properly handle failure when remote server is down

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3636 contains cbas-core commit de25d8f with commit message: MB-42037 : Properly handle failure when remote server is down
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit 8c33722 with commit message:
            MB-42037: don't encapsulate product exceptions

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 8c33722 with commit message: MB-42037 : don't encapsulate product exceptions
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit ad5211b with commit message:
            MB-42037: Rethrow error if it is already handled properly

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit ad5211b with commit message: MB-42037 : Rethrow error if it is already handled properly
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit 99142ff with commit message:
            MB-42037: Properly display error messages from remote clusters

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit 99142ff with commit message: MB-42037 : Properly display error messages from remote clusters
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9599 contains cbas-core commit de25d8f with commit message:
            MB-42037: Properly handle failure when remote server is down

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9599 contains cbas-core commit de25d8f with commit message: MB-42037 : Properly handle failure when remote server is down

              People

              Assignee:
              umang.agrawal Umang
              Reporter:
              umang.agrawal Umang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty