Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39579

[CX] link servlet answers requests before cluster state ACTIVE

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • CX Sprint 201

    Description

      Observed when running ClusterCbRemoteLinksIT single-test after a node restart was introduced- the servlet accepts requests and attempts to run them when the cluster is not yet booted, which results in strange errors, e.g.

      2020-05-27T09:18:23.271-04:00 INFO CBAS.bootstrap.MetadataBootstrap [Executor-10:1aec05372d9aeb09d8ba6c95daa4a0cd] Finished enlistment of metadata B-trees in old universe
      2020-05-27T09:18:23.494-04:00 INFO CBAS.bootstrap.MetadataBootstrap [Executor-10:1aec05372d9aeb09d8ba6c95daa4a0cd] Starting DDL recovery ...
      2020-05-27T09:18:23.636-04:00 WARN CBAS.servlet.LinkServlet [HttpExecutor(port:9601)-1] No file for Index Metadata.Link
      2020-05-27T09:18:23.638-04:00 WARN CBAS.server.AbstractServlet [HttpExecutor(port:9601)-1] Unhandled exception
      org.apache.hyracks.algebricks.common.exceptions.AlgebricksException: No file for Index Metadata.Link
      	at org.apache.asterix.metadata.MetadataNode.searchIndex(MetadataNode.java:1294) ~[asterix-metadata.jar:0.9.5-SNAPSHOT]
      	at org.apache.asterix.metadata.MetadataNode.getEntities(MetadataNode.java:272) ~[asterix-metadata.jar:0.9.5-SNAPSHOT]
      	at org.apache.asterix.metadata.MetadataNode.getEntities(MetadataNode.java:321) ~[asterix-metadata.jar:0.9.5-SNAPSHOT]
      	at org.apache.asterix.metadata.MetadataManager.getEntities(MetadataManager.java:982) ~[asterix-metadata.jar:0.9.5-SNAPSHOT]
      	at com.couchbase.analytics.metadata.CBMetadataUtil.getLink(CBMetadataUtil.java:100) ~[cbas-connector.jar:<dev build>]
      	at com.couchbase.analytics.lang.LinkDdlStatement.transactionallyHandle(LinkDdlStatement.java:130) ~[cbas-connector.jar:<dev build>]
      	at com.couchbase.analytics.lang.LinkDdlStatement.doHandle(LinkDdlStatement.java:103) ~[cbas-connector.jar:<dev build>]
      	at com.couchbase.analytics.servlet.LinkServlet.createOrAlter(LinkServlet.java:164) ~[cbas-server.jar:<dev build>]
      	at com.couchbase.analytics.servlet.LinkServlet.post(LinkServlet.java:68) ~[cbas-server.jar:<dev build>]
      	at org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:93) [hyracks-http.jar:0.3.5-SNAPSHOT]
      	at com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:81) [cbas-server.jar:<dev build>]
      	at org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:83) [hyracks-http.jar:0.3.5-SNAPSHOT]
      	at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:68) [hyracks-http.jar:0.3.5-SNAPSHOT]
      	at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37) [hyracks-http.jar:0.3.5-SNAPSHOT]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
      	at java.lang.Thread.run(Thread.java:834) [?:?]
      2020-05-27T09:18:23.660-04:00 INFO CBAS.bootstrap.MetadataBootstrap [Executor-10:1aec05372d9aeb09d8ba6c95daa4a0cd] Completed DDL recovery.
      2020-05-27T09:18:23.660-04:00 INFO CBAS.context.DatasetMemoryManager [Executor-10:1aec05372d9aeb09d8ba6c95daa4a0cd] Reserved(1048576) for dataset(53)
      

      The servlet should answer 503 until the cluster is ACTIVE, I would think.

      Attachments

        For Gerrit Dashboard: MB-39579
        # Subject Branch Project Status CR V

        Activity

          People

            michael.blow Michael Blow
            michael.blow Michael Blow
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty