Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58867

[Analytics] Unable to connect to Analytics Service

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • 7.2.2
    • analytics
    • Operating system : SUSE Linux Enterprise 12
      Couchbase Enterprise Edition build 7.2.2-6401

    Description

      Environment

       

      Operating system : SUSE Linux Enterprise 12
      		 
      Couchbase Enterprise Edition build 7.2.1-5932

       

      This issue was seen only in SUSE Linux Enterprise 12. Did

      not observe the issue in other operating systems like Debian, Ubuntu, Red Hat Linux among others.

       Steps to reproduce

      1. Created a 4 node cluster with the following combination
        1. ec2-35-153-143-50.compute-1.amazonaws.com - cbas
        2. ec2-44-210-150-34.compute-1.amazonaws.com - kv
        3. ec2-44-211-84-44.compute-1.amazonaws.com - index, kv, n1ql
        4. ec2-3-235-143-155.compute-1.amazonaws.com - cbas
      2. Created 2 couchstore buckets with 30000 items in each
      3. Created 5 dataverses and 10 datasets per dataverse.

      Unable to connect to analytics service on ec2-35-153-143-50.compute-1.amazonaws.com

      Observing errors in ns_server.analytics_error.log

      2023-09-28T13:42:18.120+00:00 ERRO CBAS.server.HttpRequestHandler [HighPriorityHttpExecutor(port:8095)-5] Failure handling HTTP Requestorg.apache.hyracks.api.exceptions.HyracksDataException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1    at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:49) ~[hyracks-api-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.api.util.InvokeUtil.tryIoWithCleanups(InvokeUtil.java:284) ~[hyracks-api-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.ChunkedResponse.close(ChunkedResponse.java:121) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:88) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:68) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]    at java.lang.Thread.run(Thread.java:833) ~[?:?]Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:148) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]    at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101) ~[netty-buffer-4.1.94.Final.jar:4.1.94.Final]    at org.apache.hyracks.http.server.ChunkedNettyOutputStream.close(ChunkedNettyOutputStream.java:89) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.api.util.InvokeUtil.tryIoWithCleanups(InvokeUtil.java:262) ~[hyracks-api-7.2.2-6401.jar:7.2.2-6401]    ... 8 more

      Observing warn in ns_server.analytics.log

       

      2023-09-28T13:42:18.119+00:00 WARN CBAS.server.QueryServiceServlet [HighPriorityHttpExecutor(port:8095)-5] handleException: unexpected exception: <ud>{"host":"127.0.0.1:8095","path":"/analytics/service","statement":"select * from `Metadata`.`Link`","pretty":false,"mode":"immediate","clientContextID":null,"clientType":"ASTERIX","dataverse":null,"format":"CLEAN_JSON","timeout":9223372036854775807,"maxResultReads":1,"planFormat":"JSON","expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"profile":"counts","signature":true,"multiStatement":false,"parseOnly":false,"readOnly":true,"maxWarnings":0,"sqlCompat":false,"scanConsistency":null,"scanWait":null}</ud>java.lang.InterruptedException: null    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386) ~[?:?]    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) ~[?:?]    at org.apache.asterix.api.http.server.NCQueryServiceServlet.executeStatement(NCQueryServiceServlet.java:98) ~[asterix-app-7.2.2-6401.jar:7.2.2-6401]    at com.couchbase.analytics.servlet.AnalyticsQueryServiceServlet.executeStatement(AnalyticsQueryServiceServlet.java:97) ~[cbas-server-7.2.2-6401.jar:7.2.2-6401]    at org.apache.asterix.api.http.server.QueryServiceServlet.handleRequest(QueryServiceServlet.java:305) ~[asterix-app-7.2.2-6401.jar:7.2.2-6401]    at org.apache.asterix.api.http.server.QueryServiceServlet.get(QueryServiceServlet.java:134) ~[asterix-app-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:90) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:93) ~[cbas-server-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:83) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:68) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37) ~[hyracks-http-7.2.2-6401.jar:7.2.2-6401]    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]    at java.lang.Thread.run(Thread.java:833) ~[?:?]2023-09-28T13:42:18.120+00:00 WARN CBAS.server.QueryServiceServlet [HighPriorityHttpExecutor(port:8095)-5] Error flushing output writer

       

      This issue was seen when trying to validate MB-58559. Could be related.

       

       

      TAF Script to reproduce

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/suse12-p0-os_certify-vset00-00-analytics/testexec.2233.ini replicas=0,GROUP=P0;durability,kv_quota_percent=70,get-cbcollect-info=True,get-cbcollect-info=False,get-cbcollect-info=True,hostname=true,upgrade_version=7.2.1-5932,sirius_url=http://172.23.120.103:4000 -t cbas.cbas_collection_functional_test.CBASDatasetsAndCollections.test_create_drop_datasets,cluster_kv_infra=bkt_spec,bucket_spec=analytics.multi_bucket,cbas_spec=local_datasets,no_of_dv=5,no_of_threads=20,ds_per_dv=10,GROUP=P0;durability,nodes_init=4,services_init=kv:n1ql-kv-cbas-cbas'

       

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-58870 io.netty.util.IllegalReferenceCountException due to duplicate buffer release on flush error

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty