Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56682

Analytics queries are timing out while the analytics nodes are stuck in rebalance

    XMLWordPrintable

Details

    Description

       Scenario -
      1. Have a capella GCP cluster with 3 KV, 3 N1ql, 3 CBAS and 3 Index nodes.
      2. KV and CBAS node has 300 GB disk on each node.
      3. Load 2 million docs on KV.
      4. Create 10 datasets on cbas. So the total docs ingested by CBAS is 20 million.
      5. Once ingestion is complete start parallel query execution on CBAS.
      6. Start scaling of cluster.
       
      Cluster version Enterprise Edition 7.1.4 build 3630

      Following logs are from automation script -

       

      2023-04-28 02:52:54,224 | infra | INFO    | pool-5-thread-7 | [task:call:300] Scaling: Status==scaling, State==Specifying settings for the cluster, Progress==90%
      		 
      16238 CBAS queries submitted, 0 failed,                         7960 passed, 0 rejected,                         0 cancelled, 0 timeout 8267 errored
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:53530","lastDispatchedTo":"svc-a-node-019.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754146,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"4d145839-cad3-4865-b2a7-56976e8ecca3","priority":0,"statement":"SELECT * from ds1 where `type` is not null limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75002416}}
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:53530","lastDispatchedTo":"svc-a-node-019.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754146,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"4d145839-cad3-4865-b2a7-56976e8ecca3","priority":0,"statement":"SELECT * from ds1 where `type` is not null limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75002416}}
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:36474","lastDispatchedTo":"svc-a-node-012.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754469,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"a610e19a-c0b6-4ba8-a389-740a8d5a3ba2","priority":0,"statement":"select avg(price) as AvgPrice, min(price) as MinPrice, max(price) as MaxPrice from ds5 where free_breakfast=True and free_parking=True and price is not null and array_count(public_likes)>5 and `type`='Hotel' group by country limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75001251}}
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:36474","lastDispatchedTo":"svc-a-node-012.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754469,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"a610e19a-c0b6-4ba8-a389-740a8d5a3ba2","priority":0,"statement":"select avg(price) as AvgPrice, min(price) as MinPrice, max(price) as MaxPrice from ds5 where free_breakfast=True and free_parking=True and price is not null and array_count(public_likes)>5 and `type`='Hotel' group by country limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75001251}}
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:57404","lastDispatchedTo":"svc-a-node-011.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754603,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"35b8dca5-3721-47f4-8bbe-826e049c8ee0","priority":0,"statement":"select avg(price) as AvgPrice, min(price) as MinPrice, max(price) as MaxPrice from ds5 where free_breakfast=True and free_parking=True and price is not null and array_count(public_likes)>5 and `type`='Hotel' group by country limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75010342}}
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:53712","lastDispatchedTo":"svc-a-node-016.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754604,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"5fb05158-aee7-4c75-ae85-849da33eb2de","priority":0,"statement":"SELECT * from ds7 where `type` is not null limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75010460}}
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:52722","lastDispatchedTo":"svc-a-node-019.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754605,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"8f59eaa3-387c-4439-b508-77534fa1bef1","priority":0,"statement":"SELECT * from ds1 where `type` is not null limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75010443}}
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:57404","lastDispatchedTo":"svc-a-node-011.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754603,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"35b8dca5-3721-47f4-8bbe-826e049c8ee0","priority":0,"statement":"select avg(price) as AvgPrice, min(price) as MinPrice, max(price) as MaxPrice from ds5 where free_breakfast=True and free_parking=True and price is not null and array_count(public_likes)>5 and `type`='Hotel' group by country limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75010342}}
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:53712","lastDispatchedTo":"svc-a-node-016.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754604,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"5fb05158-aee7-4c75-ae85-849da33eb2de","priority":0,"statement":"SELECT * from ds7 where `type` is not null limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75010460}}
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:52722","lastDispatchedTo":"svc-a-node-019.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754605,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"8f59eaa3-387c-4439-b508-77534fa1bef1","priority":0,"statement":"SELECT * from ds1 where `type` is not null limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75010443}}
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:34396","lastDispatchedTo":"svc-a-node-012.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754885,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"07fc5f3e-ac03-42cb-b4a8-edb79f71856a","priority":0,"statement":"WITH city_avg AS (SELECT city, AVG(price) AS avgprice FROM ds3 WHERE price IS NOT NULL GROUP BY city) SELECT h.name, h.price FROM ds3 h JOIN city_avg ON h.city = city_avg.city WHERE h.price < city_avg.avgprice AND h.price IS NOT NULL limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75009604}}
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:34396","lastDispatchedTo":"svc-a-node-012.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5754885,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"07fc5f3e-ac03-42cb-b4a8-edb79f71856a","priority":0,"statement":"WITH city_avg AS (SELECT city, AVG(price) AS avgprice FROM ds3 WHERE price IS NOT NULL GROUP BY city) SELECT h.name, h.price FROM ds3 h JOIN city_avg ON h.city = city_avg.city WHERE h.price < city_avg.avgprice AND h.price IS NOT NULL limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75009604}}
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:57942","lastDispatchedTo":"svc-a-node-011.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755006,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"d0f58a4a-5116-46a2-a6d3-d81b874b33d4","priority":0,"statement":"SELECT * from ds6 where phone like \"4%\" limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75008136}}
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:57942","lastDispatchedTo":"svc-a-node-011.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755006,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"d0f58a4a-5116-46a2-a6d3-d81b874b33d4","priority":0,"statement":"SELECT * from ds6 where phone like \"4%\" limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75008136}}
      		 
      Traceback (most recent call last):
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:53014","lastDispatchedTo":"svc-a-node-016.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755084,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"bc8f872f-4a58-46e9-81ec-f889d874977e","priority":0,"statement":"select avg(price) as AvgPrice, min(price) as MinPrice, max(price) as MaxPrice from ds5 where free_breakfast=True and free_parking=True and price is not null and array_count(public_likes)>5 and `type`='Hotel' group by country limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75008005}}
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:53014","lastDispatchedTo":"svc-a-node-016.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755084,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"bc8f872f-4a58-46e9-81ec-f889d874977e","priority":0,"statement":"select avg(price) as AvgPrice, min(price) as MinPrice, max(price) as MaxPrice from ds5 where free_breakfast=True and free_parking=True and price is not null and array_count(public_likes)>5 and `type`='Hotel' group by country limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75008005}}
      		 
      Traceback (most recent call last):
      		 
      Traceback (most recent call last):
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 219, in _run_query
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:34576","lastDispatchedTo":"svc-a-node-012.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755086,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"2cc58f27-5af3-46c7-b760-2918362c599d","priority":0,"statement":"WITH city_avg AS (SELECT city, AVG(price) AS avgprice FROM ds3 WHERE price IS NOT NULL GROUP BY city) SELECT h.name, h.price FROM ds3 h JOIN city_avg ON h.city = city_avg.city WHERE h.price < city_avg.avgprice AND h.price IS NOT NULL limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75007946}}
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
      com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:54242","lastDispatchedTo":"svc-a-node-019.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755085,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"62b9d48c-cd06-460e-abac-d60b83497974","priority":0,"statement":"SELECT COUNT(*) FILTER (WHERE free_breakfast = TRUE) AS count_free_breakfast, COUNT(*) FILTER (WHERE free_parking = TRUE) AS count_free_parking, COUNT(*) FILTER (WHERE free_breakfast = TRUE AND free_parking = TRUE) AS count_free_parking_and_breakfast FROM ds2 WHERE city LIKE 'North%' ORDER BY count_free_parking_and_breakfast DESC  limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75008094}}
      		 
          status, _, _, results, _ = self.execute_statement_on_cbas(
      		 
        File "pytests/aGoodDoctor/hostedCbas.py", line 284, in execute_statement_on_cbas
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:54242","lastDispatchedTo":"svc-a-node-019.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755085,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"62b9d48c-cd06-460e-abac-d60b83497974","priority":0,"statement":"SELECT COUNT(*) FILTER (WHERE free_breakfast = TRUE) AS count_free_breakfast, COUNT(*) FILTER (WHERE free_parking = TRUE) AS count_free_parking, COUNT(*) FILTER (WHERE free_breakfast = TRUE AND free_parking = TRUE) AS count_free_parking_and_breakfast FROM ds2 WHERE city LIKE 'North%' ORDER BY count_free_parking_and_breakfast DESC  limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75008094}}
      		 
          raise Exception(str(e))
      		 
      Exception: com.couchbase.client.core.error.AmbiguousTimeoutException: AnalyticsRequest, Reason: TIMEOUT {"cancelled":true,"completed":true,"coreId":"0xf2fa84530000003e","idempotent":false,"lastDispatchedFrom":"172.23.107.117:34576","lastDispatchedTo":"svc-a-node-012.i6cdil5sswnaiipe.sandbox.nonprod-project-avengers.com:18095","reason":"TIMEOUT","requestId":5755086,"requestType":"AnalyticsRequest","retried":0,"service":{"httpMethod":"POST","operationId":"2cc58f27-5af3-46c7-b760-2918362c599d","priority":0,"statement":"WITH city_avg AS (SELECT city, AVG(price) AS avgprice FROM ds3 WHERE price IS NOT NULL GROUP BY city) SELECT h.name, h.price FROM ds3 h JOIN city_avg ON h.city = city_avg.city WHERE h.price < city_avg.avgprice AND h.price IS NOT NULL limit 100","type":"analytics","uri":"/analytics/service"},"timeoutMs":75000,"timings":{"totalMicros":75007946}}

      Attachments

        Issue Links

          links to

          Web Link ASTERIXDB-3189

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              umang.agrawal Umang
              umang.agrawal Umang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty