Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44907

[System Test] Query service not responding on one query node

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-4626
      Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale : 2
      Iteration : 3rd

      Accessing query workbench from the UI on 172.23.108.103:8091 in the system test cluster is throwing error 504 -

      Received error 504 when contacting the query service. Try refreshing the browser, as errors may occur if the query service is still warming up or suffering a transient error.
      

      But this works when accessing query workbench through 172.23.120.245:8091.

      I am able to run queries through cbq and REST API just fine on 172.23.120.245, but not on 172.23.96.251 -

      [root@s13001-cnt7 bin]# ./cbq -e 172.23.96.251:8091 -u Administrator -p password
       Connected to : http://172.23.96.251:8091/. Type Ctrl-D or \QUIT to exit.
       
       Path to history file for the shell : /root/.cbq_history
      cbq> select count(*) from system:active_requests;
       
      [root@s13001-cnt7 bin]# ./cbq -e 172.23.120.245:8093 -u Administrator -p password
       Connected to : http://172.23.120.245:8093/. Type Ctrl-D or \QUIT to exit.
       
       Path to history file for the shell : /root/.cbq_history
      cbq> select count(*) from system:active_requests;
      {
          "requestID": "381e5e87-863c-4c0a-bbf2-8a1d83f7f26f",
          "signature": {
              "$1": "number"
          },
          "results": [
          {
              "$1": 27302
          }
          ],
          "status": "success",
          "metrics": {
              "elapsedTime": "68.134473ms",
              "executionTime": "67.927789ms",
              "resultCount": 1,
              "resultSize": 27,
              "serviceLoad": 3
          }
      }
      

      HQ-Computer:~ mihirkamdar$ curl -v http://172.23.96.251:8093/query/service -d 'statement=select 1' -u Administrator:password
      *   Trying 172.23.96.251...
      * TCP_NODELAY set
      * Connected to 172.23.96.251 (172.23.96.251) port 8093 (#0)
      * Server auth using Basic with user 'Administrator'
      > POST /query/service HTTP/1.1
      > Host: 172.23.96.251:8093
      > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
      > User-Agent: curl/7.54.0
      > Accept: */*
      > Content-Length: 18
      > Content-Type: application/x-www-form-urlencoded
      >
      * upload completely sent off: 18 out of 18 bytes
      ^C
      HQ-Computer:~ mihirkamdar$ curl -v http://172.23.120.245:8093/query/service -d 'statement=select 1' -u Administrator:password
      *   Trying 172.23.120.245...
      * TCP_NODELAY set
      * Connected to 172.23.120.245 (172.23.120.245) port 8093 (#0)
      * Server auth using Basic with user 'Administrator'
      > POST /query/service HTTP/1.1
      > Host: 172.23.120.245:8093
      > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
      > User-Agent: curl/7.54.0
      > Accept: */*
      > Content-Length: 18
      > Content-Type: application/x-www-form-urlencoded
      >
      * upload completely sent off: 18 out of 18 bytes
      < HTTP/1.1 200 OK
      < Content-Length: 254
      < Content-Type: application/json; version=7.0.0-N1QL
      < Date: Fri, 12 Mar 2021 02:52:45 GMT
      <
      {
      "requestID": "a959b40b-dc0d-4e46-a75a-473a0928ef4a",
      "signature": {"$1":"number"},
      "results": [
      {"$1":1}
      ],
      "status": "success",
      "metrics": {"elapsedTime": "662.265µs","executionTime": "557.763µs","resultCount": 1,"resultSize": 8,"serviceLoad": 3}
      }
      * Connection #0 to host 172.23.120.245 left intact

      This could be related to MB-44864. The latest cpu, heap, memory profiles and goroutine dumps are attached along with the dump of system:active_requests, system:complete_requests.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty