Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44612

Analytics crash with query service terminates client connection

    XMLWordPrintable

Details

    Description

      System test: There are 2 questions:

      a) Why did the server shutdown at this point of time.
      b) Client connection was broken, of course if the port for the communication was not available, this is expected behaviour.

      Step to run queries

      �2021-02-26 06:31:30,617 - root - INFO - CouchbaseException from Java SDK. com.couchbase.client.core.ServiceNotAvailableException: The Analytics service is not enabled or no node in the cluster supports it.
      �CouchbaseException from Java SDK. com.couchbase.client.core.ServiceNotAvailableException: The Analytics service is not enabled or no node in the cluster supports it.
      #Traceback (most recent call last):
      J  File "/AnalyticsQueryApp/Query/load_queries.py", line 1060, in <module>
          main()
      F  File "/AnalyticsQueryApp/Query/load_queries.py", line 1009, in main
      Z    queries = load.generate_queries_for_analytics(options.analytics_queries, bucket_list,
      _  File "/AnalyticsQueryApp/Query/load_queries.py", line 865, in generate_queries_for_analytics
      \    output = self.execute_statement_on_cbas(statement, pretty=True, client_context_id=None,
      Z  File "/AnalyticsQueryApp/Query/load_queries.py", line 538, in execute_statement_on_cbas
      *    raise Exception("CouchbaseException")
      Exception: CouchbaseException
      
      

      Logs in node - 172.23.120.74

      2021-02-26T00:22:31.355-08:00 ERRO CBAS.nc.NodeControllerService [ShutdownHook-a83c5d9735862de47f511c4b9eb9413d] Some jobs failed to exit, continuing with abnormal shutdown
      2021-02-26T00:22:34.677-08:00 ERRO CBAS.impl.IPCConnectionManager [IPC Network Listener Thread [/0:0:0:0:0:0:0:0:9115]] TCP read error from /0:0:0:0:0:0:0:0:9112
      java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4a35c8d4[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@7bc65160[Wrapped task = org.apache.hyracks.control.nc.task.HeartbeatAckTask@420d6731]] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1db3d2be[Shutting down, pool size = 291, active threads = 291, queued tasks = 0, completed tasks = 2806893]
              at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source) ~[?:?]
              at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) ~[?:?]
              at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:?]
              at java.util.concurrent.AbstractExecutorService.submit(Unknown Source) ~[?:?]
              at org.apache.hyracks.control.nc.NodeControllerIPCI.deliverIncomingMessage(NodeControllerIPCI.java:147) ~[hyracks-control-nc.jar:7.0.0-4547]
              at org.apache.hyracks.ipc.impl.IPCSystem.deliverIncomingMessage(IPCSystem.java:102) ~[hyracks-ipc.jar:7.0.0-4547]
              at org.apache.hyracks.ipc.impl.IPCHandle.processIncomingMessages(IPCHandle.java:189) ~[hyracks-ipc.jar:7.0.0-4547]
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.read(IPCConnectionManager.java:440) [hyracks-ipc.jar:7.0.0-4547]
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.processSelectedKeys(IPCConnectionManager.java:261) [hyracks-ipc.jar:7.0.0-4547]
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:231) [hyracks-ipc.jar:7.0.0-4547]
              at org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:213) [hyracks-ipc.jar:7.0.0-4547]
      2021-02-26T00:22:40.101-08:00 ERRO CBAS.lifecycle.LifeCycleComponentManager [ShutdownHook-a83c5d9735862de47f511c4b9eb9413d] Stopping instance
      

      • Not sure why the nodes had to shutdown.

      Attachments

        Issue Links

          Activity

            People

              ritam.sharma Ritam Sharma
              ritam.sharma Ritam Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty