Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29883

Analytic goes down if analytics service memory setting is modified after adding analytics nodes and initial analytics setup.

    XMLWordPrintable

Details

    Description

      1. Add an analytics nodes to the KV cluster. Set some memory.
      2. Setup analytics corresponding to KV buckets and make sure ingestion happens.
      3. Now go to setting and increase the analytics memory.
      4. Observe that analytics goes down.

      Analytics service returned status: 500

      Couple of observations from the logs:

      1. as soon as the memory is reconfigured there is a restart of cbas
      Logs:

      2018-05-30T05:23:53.721-07:00 INFO CBAS.dataset.DatasetPartitionWriter [Executor-59:524d0e3c96191f0861b1c5f3ab0d2754] open(0)
      [goport(/opt/couchbase/bin/cbas)] 2018/05/30 05:23:54 Timeout while flushing stderr
      2:ok,
      2018-05-30T05:23:55.160-07:00 INFO CBAS.cbas ++++++++++++ starting cbas main ++++++++++++
      2018-05-30T05:23:55.160-07:00 INFO CBAS.cbas uuid: 524d0e3c96191f0861b1c5f3ab0d2754
      2018-05-30T05:23:55.160-07:00 INFO CBAS.cbas A single io device was provided; auto-adjusting...
      2018-05-30T05:23:55.160-07:00 INFO CBAS.cbas configuring io devices based on logical CPUs: 8
      2018-05-30T05:23:55.160-07:00 INFO CBAS.cbas io devices were adjusted to a total of: 8
      

      2. Incorrect partition mapping on nodes

      2018-05-30T05:23:57.755-07:00 ERRO CBAS.control.AnalyticsDriver [main] Exiting AnalyticsDriver due to exception
      org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.asterix.common.exceptions.AsterixException: ASX4000: Partition id 2 for node 524d0e3c96191f0861b1c5f3ab0d2754 already in use by node 0cf19b03f954c859d1467a4b54d38e31
      at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:51) ~[hyracks-api.jar:5.5.0-2814]
      at org.apache.asterix.common.config.AsterixProperties.lambda$registerConfigOptions$0(AsterixProperties.java:47) ~[asterix-common.jar:5.5.0-2814]
      at org.apache.hyracks.control.common.config.ConfigManager.processConfig(ConfigManager.java:231) ~[hyracks-control-common.jar:5.5.0-2814]
      at org.apache.hyracks.control.nc.NodeControllerService.<init>(NodeControllerService.java:200) ~[hyracks-control-nc.jar:5.5.0-2814]
      at com.couchbase.analytics.control.AnalyticsDriver.createNcService(AnalyticsDriver.java:189) ~[cbas-server.jar:5.5.0-2814]
      at com.couchbase.analytics.control.AnalyticsDriver.initNcService(AnalyticsDriver.java:141) ~[cbas-server.jar:5.5.0-2814]
      at com.couchbase.analytics.control.AnalyticsDriver.main(AnalyticsDriver.java:73) [cbas-server.jar:5.5.0-2814]
      Caused by: org.apache.asterix.common.exceptions.AsterixException: ASX4000: Partition id 2 for node 524d0e3c96191f0861b1c5f3ab0d2754 already in use by node 0cf19b03f954c859d1467a4b54d38e31
      at org.apache.asterix.common.exceptions.AsterixException.create(AsterixException.java:64) ~[asterix-common.jar:5.5.0-2814]
      at org.apache.asterix.common.config.PropertiesAccessor.configureNc(PropertiesAccessor.java:128) ~[asterix-common.jar:5.5.0-2814]
      at org.apache.asterix.common.config.PropertiesAccessor.<init>(PropertiesAccessor.java:79) ~[asterix-common.jar:5.5.0-2814]
      at org.apache.asterix.common.config.PropertiesAccessor.getInstance(PropertiesAccessor.java:202) ~[asterix-common.jar:5.5.0-2814]
      at org.apache.asterix.common.config.AsterixProperties.lambda$registerConfigOptions$0(AsterixProperties.java:45) ~[asterix-common.jar:5.5.0-2814]
      ... 5 more
      2018-05-30T05:23:57.826-07:00 INFO CBAS.cbas analytics driver exited; exiting cbas wrapper
      2018-05-30T05:23:57.827-07:00 INFO CBAS.cbas ++++++++++++ exiting cbas main ++++++++++++

      And after this analytics is restarting and crashing in a infinite loop.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty