Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40176

Flex:SystemTest: Active query: QueryBleve remote client returned status: 500: when adding fts node

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.0.0
    • 6.6.0, Cheshire-Cat
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 6.6.0 build 7843
      Test: -test tests/n1ql/test_n1ql_fts_flexindex.yml -scope tests/n1ql/scope_flex_index_12nodes.yml
      Scale: 2

      Active query returned "rpc error: code = DeadlineExceeded desc = grpc_server:", when fts node is added at the same time:

       {
          "completed_requests": {
            "clientContextID": "query_thread_1",
            "elapsedTime": "4m7.050250354s",
            "errorCount": 1,
            "errors": [
              {
                "cause": "rpc error: code = DeadlineExceeded desc = grpc_server: Search searchInContext err: bleve: QueryBleve remote client returned status: 500 body: null",
                "code": 5000,
                "key": "Internal Error",
                "message": "n1fty: response_handler: stream.Recv, err "
              }
            ],
            "node": "172.23.120.73:8091",
            "phaseCounts": {
              "fetch": 3227691,
              "filter": 3025522,
              "ftsSearch": 3227678
            },
            "phaseOperators": {
              "authorize": 1,
              "fetch": 1,
              "filter": 1,
              "ftsSearch": 1
            },
            "phaseTimes": {
              "authorize": "32.384µs",
              "fetch": "3m52.495257396s",
              "filter": "54.626886468s",
              "ftsSearch": "30.674601103s",
              "instantiate": "30.596µs",
              "parse": "1.152385ms",
              "plan": "2.793536ms",
              "run": "4m7.046185971s"
            },
            "remoteAddr": "172.23.104.254:47726",
            "requestId": "5a0f0963-987f-4e36-9aab-94658d78cd84",
            "requestTime": "2020-06-26T09:49:13.294-07:00",
            "resultCount": 1,
            "resultSize": 64,
            "scanConsistency": "unbounded",
            "serviceTime": "4m7.050195507s",
            "state": "completed",
            "statement": "select result,AVG(rating),MIN(rating),MAX(rating) from `other-2` use index (using fts) where  rating >= 200 and rating <= 1000 and `type`=\"gideon\" group by result limit 1000\n",
            "userAgent": "couchbase-java-client/${project.version} (git: ${git.commit.id.describe}, core: ${git.commit.id.describe}) (Linux/3.10.0-1062.9.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_252-b09)",
            "users": "Administrator"
          }
        }
      

      Query backfill for request : 5a0f0963-987f-4e36-9aab-94658d78cd84, started at 2020-06-26T09:45:06 and ended at 2020-06-26T09:49:13.

      ns_server.query.log:_time=2020-06-26T09:45:06.257-07:00 _level=INFO _msg=response_handler: n1fty[bucket2_idx1/other-2-1593189906249059071] "5a0f0963-987f-4e36-9aab-94658d78cd84" started backfill for /opt/couchbase/var/lib/couchbase/tmp/search-results24541351064986 
      ns_server.query.log:_time=2020-06-26T09:49:13.233-07:00 _level=INFO _msg=response_handler: n1fty[bucket2_idx1/other-2-1593189906249059071] "5a0f0963-987f-4e36-9aab-94658d78cd84" finished backfill for /opt/couchbase/var/lib/couchbase/tmp/search-results24541351064986  
      

      Rebalance with adding fts node started in the test by 2020-06-26T09:24:10 and finished by 2020-06-26T09:50:00.

      [2020-06-26T09:24:01-07:00, sequoiatools/couchbase-cli:6.5:c96a4f] server-add -c 172.23.97.74:8091 --server-add https://172.23.123.26 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
      [2020-06-26T09:24:10-07:00, sequoiatools/couchbase-cli:6.5:4188b6] rebalance -c 172.23.97.74:8091 -u Administrator -p password
      [2020-06-26T09:48:54-07:00, sequoiatools/cmd:02abaf] 60
      [2020-06-26T09:50:00-07:00, sequoiatools/cmd:a1104c] 900
      

      Logs:

      cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.96.18.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1593193208/collectinfo-2020-06-26T174009-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty