Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48428

Query rest admin/stats not fully populated

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Neo
    • Fix Version/s: Neo
    • Component/s: query
    • Labels:
    • Environment:
      7.1.0-1243
    • Triage:
      Untriaged
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      While running query workload I tried to check query stats via:

      curl http://localhost:8093/admin/stats

      Looks like some data is not updated ... was specifically looking to monitor active vs queued requests.

      During tests I run multiple query thread and then check stats and also system:active_requests. 

      Here is what I see after I launch 16 query threads ... I also lower servicers setting to 4.

      system:active_requests:

      2021-09-12 15:15:46 | INFO | MainProcess | test_thread | [tuq.run_cbq_query] RUN QUERY SELECT state, node, count(*) as count FROM system:active_requests WHERE statement = 'select sleep(10000)' GROUP BY state, node
      2021-09-12 15:15:46 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+state%2C+node%2C+count%28%2A%29+as+count+FROM+system%3Aactive_requests+WHERE+statement+%3D+%27select+sleep%2810000%29%27+GROUP+BY+state%2C+node&scan_consistency=REQUEST_PLUS
      2021-09-12 15:15:46 | INFO | MainProcess | test_thread | [tuq.run_cbq_query] TOTAL ELAPSED TIME: 45.081809ms
      2021-09-12 15:15:46 | INFO | MainProcess | test_thread | [tuq_graceful_failover.test_failover] {'requestID': '90713ef2-15a1-4940-9f47-fc5afe7f1511', 'signature': {'count': 'number', 'node': 'json', 'state': 'json'}, 'results': [{'count': 16, 'node': '172.23.104.92:8091', 'state': 'running'}], 'status': 'success', 'metrics': {'elapsedTime': '45.081809ms', 'executionTime': '44.966265ms', 'resultCount': 1, 'resultSize': 58, 'serviceLoad': 1}}
      2021-09-12 15:15:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 2 secs.  ...
      
      

      and admin/stats via REST:

      2021-09-12 15:15:48 | INFO | MainProcess | test_thread | [rest_client.query_tool_stats] query n1ql stats 2021-09-12 15:15:48 | INFO | MainProcess | test_thread | [tuq_graceful_failover.test_failover] {'active_requests.count': 0, 'at_plus.count': 0, 'audit_actions.count': 0, 'audit_actions_failed.count': 0, 'audit_requests_filtered.count': 60, 'audit_requests_total.count': 60, 'cancelled.count': 0, 'deletes.count': 0, 'errors.count': 0, 'index_scans.count': 0, 'inserts.count': 0, 'invalid_requests.count': 0, 'load.value': 25, 'mutations.count': 0, 'prepared.count': 0, 'primary_scans.count': 0, 'queued_requests.count': 0, 'request_time.count': 0, 'request_timer.15m.rate': 0, 'request_timer.1m.rate': 0, 'request_timer.5m.rate': 0, 'request_timer.75%': 0, 'request_timer.95%': 0, 'request_timer.99%': 0, 'request_timer.99.9%': 0, 'request_timer.count': 0, 'request_timer.max': 0, 'request_timer.mean': 0, 'request_timer.mean.rate': 0, 'request_timer.median': 0, 'request_timer.min': 0, 'request_timer.stddev': 0, 'requests.count': 0, 'requests_1000ms.count': 0, 'requests_250ms.count': 0, 'requests_5000ms.count': 0, 'requests_500ms.count': 0, 'result_count.count': 0, 'result_size.count': 0, 'scan_plus.count': 0, 'selects.count': 0, 'service_time.count': 0, 'transaction_time.count': 0, 'transactions.count': 0, 'unbounded.count': 0, 'updates.count': 0, 'warnings.count': 0}  

       

       

       

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.1.0-1289 contains query commit a2b328d with commit message:
          MB-48428 Implement active_requests & queued_requests counters for stats REST API

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1289 contains query commit a2b328d with commit message: MB-48428 Implement active_requests & queued_requests counters for stats REST API
          Hide
          Donald.haggart Donald Haggart added a comment -

          Verification should just be observing the counters with active statements.  (Pierre Regazzoni , your sleep function could come in handy to generate long statements allowing you to see queuing.)

          Show
          Donald.haggart Donald Haggart added a comment - Verification should just be observing the counters with active statements.  ( Pierre Regazzoni , your sleep function could come in handy to generate long statements allowing you to see queuing.)
          Hide
          pierre.regazzoni Pierre Regazzoni added a comment -

          Verified on 7.1.0-1289 with 16 query thread and servicers set to 8.

          See below for info from active_requests and stats ... for stats, active_requests.count = running + queued_requests.count

          2021-09-15 15:55:33 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+state%2C+count%28%2A%29+as+count+FROM+system%3Aactive_requests+WHERE+statement+%3D+%27select+sleep%2810000%29%27+and+node+like+%27172.23.104.92%25%27+GROUP+BY+state+ORDER+BY+state&scan_consistency=REQUEST_PLUS
          2021-09-15 15:55:33 | INFO | MainProcess | test_thread | [tuq.run_cbq_query] TOTAL ELAPSED TIME: 74.705884ms
          2021-09-15 15:55:33 | INFO | MainProcess | test_thread | [tuq_graceful_failover.test_failover] [{'count': 8, 'state': 'running'}, {'count': 8, 'state': 'submitted'}]
          2021-09-15 15:55:33 | INFO | MainProcess | test_thread | [rest_client.query_tool_stats] query n1ql stats
          2021-09-15 15:55:33 | INFO | MainProcess | test_thread | [rest_client.query_tool_stats] b'{"active_requests.count":16,"at_plus.count":0,"audit_actions.count":0,"audit_actions_failed.count":0,"audit_requests_filtered.count":40,"audit_requests_total.count":40,"cancelled.count":0,"deletes.count":0,"errors.count":0,"index_scans.count":0,"inserts.count":0,"invalid_requests.count":0,"load.value":200,"mutations.count":0,"prepared.count":0,"primary_scans.count":0,"queued_requests.count":8,"request_time.count":0,"request_timer.15m.rate":0,"request_timer.1m.rate":0,"request_timer.5m.rate":0,"request_timer.75%":0,"request_timer.95%":0,"request_timer.99%":0,"request_timer.99.9%":0,"request_timer.count":0,"request_timer.max":0,"request_timer.mean":0,"request_timer.mean.rate":0,"request_timer.median":0,"request_timer.min":0,"request_timer.stddev":0,"requests.count":0,"requests_1000ms.count":0,"requests_250ms.count":0,"requests_5000ms.count":0,"requests_500ms.count":0,"result_count.count":0,"result_size.count":0,"scan_plus.count":0,"selects.count":0,"service_time.count":0,"transaction_time.count":0,"transactions.count":0,"unbounded.count":0,"updates 

          Show
          pierre.regazzoni Pierre Regazzoni added a comment - Verified on 7.1.0-1289 with 16 query thread and servicers set to 8. See below for info from active_requests and stats ... for stats , active_requests.count = running + queued_requests.count 2021 - 09 - 15 15 : 55 : 33 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+state%2C+count% 28 %2A% 29 +as+count+FROM+system%3Aactive_requests+WHERE+statement+%3D+%27select+sleep% 2810000 % 29 % 27 +and+node+like+% 27172.23 . 104.92 % 25 % 27 +GROUP+BY+state+ORDER+BY+state&scan_consistency=REQUEST_PLUS 2021 - 09 - 15 15 : 55 : 33 | INFO | MainProcess | test_thread | [tuq.run_cbq_query] TOTAL ELAPSED TIME: 74 .705884ms 2021 - 09 - 15 15 : 55 : 33 | INFO | MainProcess | test_thread | [tuq_graceful_failover.test_failover] [{ 'count' : 8 , 'state' : 'running' }, { 'count' : 8 , 'state' : 'submitted' }] 2021 - 09 - 15 15 : 55 : 33 | INFO | MainProcess | test_thread | [rest_client.query_tool_stats] query n1ql stats 2021 - 09 - 15 15 : 55 : 33 | INFO | MainProcess | test_thread | [rest_client.query_tool_stats] b'{ "active_requests.count" : 16 , "at_plus.count" : 0 , "audit_actions.count" : 0 , "audit_actions_failed.count" : 0 , "audit_requests_filtered.count" : 40 , "audit_requests_total.count" : 40 , "cancelled.count" : 0 , "deletes.count" : 0 , "errors.count" : 0 , "index_scans.count" : 0 , "inserts.count" : 0 , "invalid_requests.count" : 0 , "load.value" : 200 , "mutations.count" : 0 , "prepared.count" : 0 , "primary_scans.count" : 0 , "queued_requests.count" : 8 , "request_time.count" : 0 , "request_timer.15m.rate" : 0 , "request_timer.1m.rate" : 0 , "request_timer.5m.rate" : 0 , "request_timer.75%" : 0 , "request_timer.95%" : 0 , "request_timer.99%" : 0 , "request_timer.99.9%" : 0 , "request_timer.count" : 0 , "request_timer.max" : 0 , "request_timer.mean" : 0 , "request_timer.mean.rate" : 0 , "request_timer.median" : 0 , "request_timer.min" : 0 , "request_timer.stddev" : 0 , "requests.count" : 0 , "requests_1000ms.count" : 0 , "requests_250ms.count" : 0 , "requests_5000ms.count" : 0 , "requests_500ms.count" : 0 , "result_count.count" : 0 , "result_size.count" : 0 , "scan_plus.count" : 0 , "selects.count" : 0 , "service_time.count" : 0 , "transaction_time.count" : 0 , "transactions.count" : 0 , "unbounded.count" : 0 ,"updates

            People

            Assignee:
            pierre.regazzoni Pierre Regazzoni
            Reporter:
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty