Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37519

internal server error seen during analytics volume test

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Unknown
    • CX Sprint 183, CX Sprint 184, CX Sprint 185, CX Sprint 186, CX Sprint 187, CX Sprint 188, CX Sprint 189, CX Sprint 210, CX Sprint 211

    Description

      While running the analytics volume test I got an internal server error seen here: http://perf.jenkins.couchbase.com/job/oceanus/2688/console

      In the logs, there are several errors seen:

       

      java.lang.UnsupportedOperationException: Boot class path mechanism is not supported at sun.management.RuntimeImpl.getBootClassPath(Unknown Source) ~[?:?] at org.apache.hyracks.util.MXHelper.getBootClassPath(MXHelper.java:111) [hyracks-util.jar:6.5.1-6026] at org.apache.hyracks.control.common.controllers.NodeRegistration.<init>(NodeRegistration.java:102) [hyracks-control-common.jar:6.5.1-6026] at org.apache.hyracks.control.nc.NodeControllerService.initNodeControllerState(NodeControllerService.java:339) [hyracks-control-nc.jar:6.5.1-6026] at org.apache.hyracks.control.nc.NodeControllerService.start(NodeControllerService.java:304) [hyracks-control-nc.jar:6.5.1-6026] at com.couchbase.analytics.control.AnalyticsDriver.startService(AnalyticsDriver.java:134) [cbas-server.jar:6.5.1-6026] at com.couchbase.analytics.control.AnalyticsDriver.main(AnalyticsDriver.java:105)[cbas-server.jar:6.5.1-6026] 2020-01-14T15:04:17.070-08:00 WARN CBAS.runtime.BucketOperatorNodePushable [Executor-17:f561f202205a6df0138f7163f3bfeccc:JID:2.220:TAID:TID:ANID:ODID:0:0:13:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.bucket-1(CouchbaseMetadataExtension))[13]:BucketOperatorDescriptor] Failure during data ingestion java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) ~[?:?] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) ~[?:?] at com.couchbase.analytics.adapter.CouchbaseConnector.pollNextMessage(CouchbaseConnector.java:469) ~[cbas-connector.jar:6.5.1-6026] at com.couchbase.analytics.adapter.CouchbaseConnector.take(CouchbaseConnector.java:446) ~[cbas-connector.jar:6.5.1-6026] at com.couchbase.analytics.adapter.CouchbaseConnector.next(CouchbaseConnector.java:434) ~[cbas-connector.jar:6.5.1-6026] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.next(FeedRecordDataFlowController.java:134) ~[asterix-external-data.jar:6.5.1-6026] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:82) ~[asterix-external-data.jar:6.5.1-6026] at org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:38) ~[asterix-external-data.jar:6.5.1-6026] at com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:52) ~[cbas-connector.jar:6.5.1-6026] at org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:102) ~[asterix-active.jar:6.5.1-6026] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:228) ~[hyracks-api.jar:6.5.1-6026] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?]org.apache.hyracks.api.exceptions.HyracksDataException: HYR0115: Local network error at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:60) ~[hyracks-api.jar:6.5.1-6026] at org.apache.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender(NonDeterministicChannelReader.java:115) ~[hyracks-dataflow-std.jar:6.5.1-6026] at org.apache.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDeterministicFrameReader.java:43) ~[hyracks-dataflow-std.jar:6.5.1-6026] at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:416) ~[hyracks-control-nc.jar:6.5.1-6026] at org.apache.hyracks.control.nc.Task.run(Task.java:354) [hyracks-control-nc.jar:6.5.1-6026] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Suppressed: org.apache.hyracks.api.exceptions.HyracksDataException: java.lang.InterruptedException at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:51) ~[hyracks-api.jar:6.5.1-6026] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:251) ~[hyracks-api.jar:6.5.1-6026] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:96) ~[hyracks-api.jar:6.5.1-6026] at org.apache.hyracks.control.nc.Task.run(Task.java:364) [hyracks-control-nc.jar:6.5.1-6026] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(Unknown Source) ~[?:?] at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:240) ~[hyracks-api.jar:6.5.1-6026] ... 5 more[Executor-17:f561f202205a6df0138f7163f3bfeccc:JID:2.220:TAID:TID:ANID:ODID:0:0:13:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.bucket-1(CouchbaseMetadataExtension))[13]:BucketOperatorDescriptor] Failure while operating a feed source java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source) ~[?:?] at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) ~[?:?] at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) ~[?:?] at com.couchbase.analytics.adapter.CouchbaseConnector.pollNextMessage(CouchbaseConnector.java:469) ~[cbas-connector.jar:6.5.1-6026] at com.couchbase.analytics.adapter.CouchbaseConnector.take(CouchbaseConnector.java:446) ~[cbas-connector.jar:6.5.1-6026] at com.couchbase.analytics.adapter.CouchbaseConnector.next(CouchbaseConnector.java:434) ~[cbas-connector.jar:6.5.1-6026] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.next(FeedRecordDataFlowController.java:134) ~[asterix-external-data.jar:6.5.1-6026] at org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:82) ~[asterix-external-data.jar:6.5.1-6026] at org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:38) ~[asterix-external-data.jar:6.5.1-6026] at com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:52) ~[cbas-connector.jar:6.5.1-6026] at org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:102) ~[asterix-active.jar:6.5.1-6026] at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:228) ~[hyracks-api.jar:6.5.1-6026] at java.util.concurrent.FutureTask.run(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?] at java.lang.Thread.run(Unknown Source) [?:?]
      

       

      Here are the logs:

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-14T233814-ns_1%40172.23.96.205.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-14T233814-ns_1%40172.23.96.5.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-14T233814-ns_1%40172.23.96.57.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-14T233814-ns_1%40172.23.96.7.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-14T233814-ns_1%40172.23.96.8.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-01-14T233814-ns_1%40172.23.96.9.zip

       

      I will upload a picture of the cluster configuration. The data is only 215GB. Server error was seen when running this query:

      WITH customer_total_return AS (SELECT sr.sr_customer_sk AS ctr_customer_sk, sr.sr_store_sk AS ctr_store_sk, SUM(sr.sr_return_amt) AS ctr_total_return FROM date_dim dd, store_returns sr WHERE tostring(dd.d_date_sk) /*+ indexnl */ = sr.sr_returned_date_sk AND dd.d_year = 2000 GROUP BY sr.sr_customer_sk, sr.sr_store_sk) SELECT c.c_customer_id FROM (SELECT ctr1.ctr_store_sk, ctr1.ctr_customer_sk FROM customer_total_return ctr1 WHERE ctr1.ctr_total_return > (SELECT VALUE AVG(ctr2.ctr_total_return) * 1.2 FROM customer_total_return ctr2 WHERE ctr1.ctr_store_sk = ctr2.ctr_store_sk)[0]) ctr2 JOIN store s ON ctr2.ctr_store_sk /*+ hash-bcast */ = s.s_store_sk JOIN customer c ON tostring(ctr2.ctr_customer_sk) /*+ indexnl */ = c.c_customer_sk WHERE s.s_state= "TN" ORDER BY c.c_customer_id LIMIT 100;
      

       

      Build was 6.5.1-6026

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            bo-chun.wang Bo-Chun Wang
            korrigan.clark Korrigan Clark (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty