Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42305

[CX] Remote link authentication waiting indefinitely on fallback to SCRAM-SHA

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • CX Sprint 223, CX Sprint 236, CX Sprint 238, CX Sprint 239

    Description

      After falling back to SCRAM-SHA, the remote link authentication waited indefinitely for a response from the remote cluster as can be seen in the stacktrace below:

      {
        "id": 246,
        "locked_synchronizers": [
          {
            "className": "java.util.concurrent.ThreadPoolExecutor$Worker",
            "identityHashCode": 1274906177
          },
          {
            "className": "java.util.concurrent.locks.ReentrantReadWriteLock$FairSync",
            "identityHashCode": 275966571
          }
        ],
        "name": "HttpExecutor(port:9111)-15",
        "stack": [
          "java.base@11.0.7/java.net.SocketInputStream.socketRead0(Native Method)",
          "java.base@11.0.7/java.net.SocketInputStream.socketRead(Unknown Source)",
          "java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)",
          "java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)",
          "app//org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)",
          "app//org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)",
          "app//org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)",
          "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)",
          "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)",
          "app//org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)",
          "app//org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)",
          "app//org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)",
          "app//org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)",
          "app//org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)",
          "app//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)",
          "app//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)",
          "app//org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)",
          "app//org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)",
          "app//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)",
          "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)",
          "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)",
          "app//com.couchbase.analytics.auth.CbRemoteLinkHelper.executeScramSha(CbRemoteLinkHelper.java:150)",
          "app//com.couchbase.analytics.auth.CbRemoteLinkHelper.execute(CbRemoteLinkHelper.java:109)",
          "app//com.couchbase.analytics.bootstrap.NsServerHelper.authenticate(NsServerHelper.java:516)",
          "app//com.couchbase.analytics.bootstrap.NsServerHelper.fallbackToScramSha(NsServerHelper.java:464)",
          "app//com.couchbase.analytics.bootstrap.NsServerHelper.authenticate(NsServerHelper.java:452)",
          "app//com.couchbase.analytics.metadata.CbRemoteLinkReference.authenticate(CbRemoteLinkReference.java:83)",
          "app//com.couchbase.analytics.lang.CreateLinkStatement.doOperation(CreateLinkStatement.java:69)",
          "app//com.couchbase.analytics.lang.LinkDdlStatement.transactionallyHandle(LinkDdlStatement.java:132)",
          "app//com.couchbase.analytics.lang.LinkDdlStatement.doHandle(LinkDdlStatement.java:102)",
          "app//com.couchbase.analytics.servlet.LinkServlet.createOrAlter(LinkServlet.java:196)",
          "app//com.couchbase.analytics.servlet.LinkServlet.lambda$post$0(LinkServlet.java:85)",
          "app//com.couchbase.analytics.servlet.LinkServlet$$Lambda$876/0x0000000800ab5440.run(Unknown Source)",
          "app//com.couchbase.analytics.servlet.LinkServlet.handleDdlRequest(LinkServlet.java:107)",
          "app//com.couchbase.analytics.servlet.LinkServlet.post(LinkServlet.java:85)",
          "app//org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:93)",
          "app//com.couchbase.analytics.servlet.LinkServlet.handle(LinkServlet.java:79)",
          "app//com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:82)",
          "app//org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:83)",
          "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:68)",
          "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37)",
          "java.base@11.0.7/java.util.concurrent.FutureTask.run(Unknown Source)",
          "java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
          "java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
          "java.base@11.0.7/java.lang.Thread.run(Unknown Source)"
        ],
        "state": "RUNNABLE"
      }
      

      Because this operation is performed while holding the connection lock, all other operations (e.g. DDLs\DMLs) that require the connection lock were waiting forever as well.

      Attachments

        Issue Links

          Activity

            People

              umang.agrawal Umang
              murtadha.hubail Murtadha Hubail
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty