Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44864

[System Test] 27K queries in closed state in system:active_requests

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-4626
      Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale : 2
      Iteration : 1st

      The query select count from system:active_requests where state=="closed" shows 27099 queries. Some of them are there since 32hrs.

      cbq> select statement,requestTime,elapsedTime,state from system:active_requests where state=="closed" limit 5;
      {
          "requestID": "9511f356-1fc6-4924-9821-3be705873186",
          "signature": {
              "elapsedTime": "json",
              "requestTime": "json",
              "state": "json",
              "statement": "json"
          },
          "results": [
          {
              "elapsedTime": "32h3m45.231338861s",
              "requestTime": "2021-03-09T01:31:41.305-08:00",
              "state": "closed",
              "statement": "select * from `default` where result=\"SUCCESS\" limit 20"
          },
          {
              "elapsedTime": "32h3m10.075343787s",
              "requestTime": "2021-03-09T01:32:16.462-08:00",
              "state": "closed",
              "statement": "select * from `default` where result=\"SUCCESS\" limit 20"
          },
          {
              "elapsedTime": "31h33m2.36324449s",
              "requestTime": "2021-03-09T02:02:24.174-08:00",
              "state": "closed",
              "statement": "select * from `default` where result=\"SUCCESS\" limit 20"
          },
          {
              "elapsedTime": "22h58m37.422609497s",
              "requestTime": "2021-03-09T10:36:49.116-08:00",
              "state": "closed",
              "statement": "select * from `default` where result=\"SUCCESS\" limit 20"
          },
          {
              "elapsedTime": "22h55m18.086693803s",
              "requestTime": "2021-03-09T10:40:08.453-08:00",
              "state": "closed",
              "statement": "select * from `default` where result=\"SUCCESS\" limit 20"
          }
          ],
          "status": "success",
          "metrics": {
              "elapsedTime": "55.896692ms",
              "executionTime": "55.726685ms",
              "resultCount": 5,
              "resultSize": 1081,
              "serviceLoad": 15
          }
      }
      

      It needs to be investigated what is causing such a large number of queries to be stuck like this, are they holding up any resources etc.

      Query service is responding fine - no issues seen in the functioning as such.

      I have attached the cpu, heap and memory profiles along with dump of active and completed requests for both the query nodes - 172.23.120.245 and 172.23.96.251

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          marco.greco Marco Greco added a comment -

          Perfect, I have what I need...

          marco.greco Marco Greco added a comment - Perfect, I have what I need...

          The fix is  tested by QE and ready to be merged today in CC.

          kamini.jagtiani Kamini Jagtiani (Inactive) added a comment - The fix is  tested by QE and ready to be merged today in CC.

          Build couchbase-server-6.6.2-9566 contains query commit ea8f6e0 with commit message:
          MB-44864 terminate keep alive operators on closed connection

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9566 contains query commit ea8f6e0 with commit message: MB-44864 terminate keep alive operators on closed connection

          Build couchbase-server-7.0.0-4695 contains query commit bd16c71 with commit message:
          MB-44864 terminate keep alive operators on closed connection

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4695 contains query commit bd16c71 with commit message: MB-44864 terminate keep alive operators on closed connection

          Verified on the system test run with 6.6.2-9576.

          mihir.kamdar Mihir Kamdar (Inactive) added a comment - Verified on the system test run with 6.6.2-9576.

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty