Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.6.0
-
Untriaged
-
1
-
Unknown
-
CX Sprint 223, CX Sprint 236, CX Sprint 238, CX Sprint 239
Description
After falling back to SCRAM-SHA, the remote link authentication waited indefinitely for a response from the remote cluster as can be seen in the stacktrace below:
{
|
"id": 246,
|
"locked_synchronizers": [
|
{
|
"className": "java.util.concurrent.ThreadPoolExecutor$Worker",
|
"identityHashCode": 1274906177
|
},
|
{
|
"className": "java.util.concurrent.locks.ReentrantReadWriteLock$FairSync",
|
"identityHashCode": 275966571
|
}
|
],
|
"name": "HttpExecutor(port:9111)-15",
|
"stack": [
|
"java.base@11.0.7/java.net.SocketInputStream.socketRead0(Native Method)",
|
"java.base@11.0.7/java.net.SocketInputStream.socketRead(Unknown Source)",
|
"java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)",
|
"java.base@11.0.7/java.net.SocketInputStream.read(Unknown Source)",
|
"app//org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)",
|
"app//org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)",
|
"app//org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)",
|
"app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)",
|
"app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)",
|
"app//org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)",
|
"app//org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)",
|
"app//org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)",
|
"app//org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)",
|
"app//org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)",
|
"app//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)",
|
"app//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)",
|
"app//org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)",
|
"app//org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)",
|
"app//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)",
|
"app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)",
|
"app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)",
|
"app//com.couchbase.analytics.auth.CbRemoteLinkHelper.executeScramSha(CbRemoteLinkHelper.java:150)",
|
"app//com.couchbase.analytics.auth.CbRemoteLinkHelper.execute(CbRemoteLinkHelper.java:109)",
|
"app//com.couchbase.analytics.bootstrap.NsServerHelper.authenticate(NsServerHelper.java:516)",
|
"app//com.couchbase.analytics.bootstrap.NsServerHelper.fallbackToScramSha(NsServerHelper.java:464)",
|
"app//com.couchbase.analytics.bootstrap.NsServerHelper.authenticate(NsServerHelper.java:452)",
|
"app//com.couchbase.analytics.metadata.CbRemoteLinkReference.authenticate(CbRemoteLinkReference.java:83)",
|
"app//com.couchbase.analytics.lang.CreateLinkStatement.doOperation(CreateLinkStatement.java:69)",
|
"app//com.couchbase.analytics.lang.LinkDdlStatement.transactionallyHandle(LinkDdlStatement.java:132)",
|
"app//com.couchbase.analytics.lang.LinkDdlStatement.doHandle(LinkDdlStatement.java:102)",
|
"app//com.couchbase.analytics.servlet.LinkServlet.createOrAlter(LinkServlet.java:196)",
|
"app//com.couchbase.analytics.servlet.LinkServlet.lambda$post$0(LinkServlet.java:85)",
|
"app//com.couchbase.analytics.servlet.LinkServlet$$Lambda$876/0x0000000800ab5440.run(Unknown Source)",
|
"app//com.couchbase.analytics.servlet.LinkServlet.handleDdlRequest(LinkServlet.java:107)",
|
"app//com.couchbase.analytics.servlet.LinkServlet.post(LinkServlet.java:85)",
|
"app//org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:93)",
|
"app//com.couchbase.analytics.servlet.LinkServlet.handle(LinkServlet.java:79)",
|
"app//com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:82)",
|
"app//org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:83)",
|
"app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:68)",
|
"app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37)",
|
"java.base@11.0.7/java.util.concurrent.FutureTask.run(Unknown Source)",
|
"java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)",
|
"java.base@11.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)",
|
"java.base@11.0.7/java.lang.Thread.run(Unknown Source)"
|
],
|
"state": "RUNNABLE"
|
}
|
Because this operation is performed while holding the connection lock, all other operations (e.g. DDLs\DMLs) that require the connection lock were waiting forever as well.
Attachments
Issue Links
For Gerrit Dashboard: MB-42305 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
139210,3 | MB-42305: use socket timeout for ns_server http connections | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
139390,3 | Revert "MB-42305: use socket timeout for ns_server http connections" | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
145880,4 | MB-42305: use socket timeout for ns_server http connections | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
147093,3 | MB-42305: use socket timeout for remote link https connections | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
147094,2 | MB-42305: s/HttpClient/HttpClientProvider/ | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
147182,4 | MB-42305: close expired https clients, minor cleanup | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |
147733,3 | MB-42305: synchronize https client cache modifications | mad-hatter | cbas-core | Status: MERGED | +2 | +1 |