Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31026

Longevity Test: Few queries failed during rebalance as the cluster state went into SHUTING_DOWN, UNUSABLE.

    XMLWordPrintable

Details

    Description

      In the longevity test some queries are failing because the cbas cluster goes into UNSTABLE state.

      Here is the change the cbas topology in the test:
      1. There are two analytics nodes to start the cluster:

      ######  cbas : 2 ===== > [172.23.96.145:8091 172.23.99.25:8091]  ###########

      2. At this time a swap rebalance is done for analytics:

      [2018-08-21T12:01:32-07:00
      		 
      ######  cbas : 2 ===== > [172.23.108.104:8091 172.23.96.145:8091]  ###########
      		 
      [2018-08-21T12:12:35-07:00

      3. Again swap rebalance is performed here for analytics:

      [2018-08-21T16:53:24-07:00
      		 
      ######  cbas : 2 ===== > [172.23.108.104:8091 172.23.97.239:8091]  ###########
      		 
      [2018-08-21T17:24:14-07:00

      At this Step few of the queries got failed:

      analytics_warn.log:2018-08-21T17:06:21.324-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-4] *handleException: ASX0032: Cannot execute request, cluster is SHUTTING_DOWN:* {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds2 where duration between 3009 and 3010 and profile is not missing;","pretty":true,"mode":null,"clientContextID":"query_thread_48772","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T17:06:23.471-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-7] *handleException: ASX0032: Cannot execute request, cluster is SHUTTING_DOWN:* {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds1 where duration between 3009 and 3010 and profile is not missing and array_length(profile.friends) > 5 limit 100;","pretty":true,"mode":null,"clientContextID":"query_thread_48775","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T17:06:49.959-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-0] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result = \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3009 and 3010 UNION ALL SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result != \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3010 and 3012;","pretty":true,"mode":null,"clientContextID":"query_thread_48785","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T17:21:00.099-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-0] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"select sum(friends.num_friends) from (select array_length(profile.friends) as num_friends from ds3) as friends;","pretty":true,"mode":null,"clientContextID":"query_thread_49000","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T17:21:04.178-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-1] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds2 where duration between 3009 and 3010 and profile is not missing;","pretty":true,"mode":null,"clientContextID":"query_thread_49002","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T17:21:08.291-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-2] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds1 where duration between 3009 and 3010 and profile is not missing and array_length(profile.friends) > 5 limit 100;","pretty":true,"mode":null,"clientContextID":"query_thread_49004","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:27.748-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-0] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds2 where duration between 3009 and 3010 and profile is not missing;","pretty":true,"mode":null,"clientContextID":"query_thread_50342","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:27.758-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-1] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds2 where duration between 3009 and 3010 and profile is not missing;","pretty":true,"mode":null,"clientContextID":"query_thread_50341","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:27.759-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-2] handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds1 where duration between 3009 and 3010 and profile is not missing and array_length(profile.friends) > 5 limit 100;","pretty":true,"mode":null,"clientContextID":"query_thread_50340","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:28.221-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-5] *handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: *{"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds2 where duration between 3009 and 3010 and profile is not missing;","pretty":true,"mode":null,"clientContextID":"query_thread_50344","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:28.223-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-4] *handleException: ASX0032: Cannot execute request, cluster is UNUSABLE:* {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result = \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3009 and 3010 UNION ALL SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result != \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3010 and 3012;","pretty":true,"mode":null,"clientContextID":"query_thread_50345","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:28.224-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-3] *handleException: ASX0032: Cannot execute request, cluster is UNUSABLE:* {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as bucketName, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds1 where duration between 3009 and 3010 and profile is not missing and array_length(profile.friends) > 5 limit 100;","pretty":true,"mode":null,"clientContextID":"query_thread_50343","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:30.214-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-7] *handleException: ASX0032: Cannot execute request, cluster is UNUSABLE:* {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result = \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3009 and 3010 UNION ALL SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result != \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3010 and 3012;","pretty":true,"mode":null,"clientContextID":"query_thread_50347","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:30.215-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-6] *handleException: ASX0032: Cannot execute request, cluster is UNUSABLE:* {"host":"172.23.97.239:8095","path":"/query/service","statement":"SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result = \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3009 and 3010 UNION ALL SELECT name as id, result as Result, `type` as `Type`, array_length(profile.friends) as num_friends FROM  ds4 where result != \\\"SUCCESS\\\" and profile is not missing and array_length(profile.friends) = 5 and duration between 3010 and 3012;","pretty":true,"mode":null,"clientContextID":"query_thread_50346","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}
      		 
       
      		 
      analytics_warn.log:2018-08-21T18:13:30.215-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-8] *handleException: ASX0032: Cannot execute request, cluster is UNUSABLE: *{"host":"172.23.97.239:8095","path":"/query/service","statement":"select sum(friends.num_friends) from (select array_length(profile.friends) as num_friends from ds3) as friends;","pretty":true,"mode":null,"clientContextID":"query_thread_50348","format":null,"timeout":"300s","maxResultReads":null,"planFormat":null,"expressionTree":false,"rewrittenExpressionTree":false,"logicalPlan":false,"optimizedLogicalPlan":false,"job":false,"signature":true,"multiStatement":false}

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty