Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44886

[System Test][ERRO] grpc_client: search err: rpc error: code = Unavailable

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0-4655

      Test: -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml

      Test cycle : 1

      Below query failed during longevity test with below errors:

      2021-03-10 17:59:30,928 - ftsindexmanager - INFO - --------------- Running query on bucket_bucket4_idx_uivgw-Overall-public_likes-free_parking field {'name': 'reviews.ratings.Overall', 'type': 'number', 'is_nested_object': True, 'field_code': 'Overall', 'queries': [{'query': 'reviews.ratings.Overall:3'}, {'min': 1, 'max': 1000, 'inclusive_min': False, 'inclusive_max': False, 'field': 'reviews.ratings.Overall'}], 'flex_queries': ['ANY v in reviews.ratings.Overall SATISFIES v = 3 END']} ---------------
      2021-03-10 17:59:30,928 - ftsindexmanager - INFO - URI : /api/index/bucket_bucket4_idx_uivgw-Overall-public_likes-free_parking/query body : {'explain': True, 'query': {'min': 1, 'max': 1000, 'inclusive_min': False, 'inclusive_max': False, 'field': 'reviews.ratings.Overall'}, 'size': 17, 'from': 7788, 'ctl': {'timeout': 120000}}
      2021-03-10 17:59:31,092 - ftsindexmanager - INFO - Index name: bucket_bucket4_idx_uivgw-Overall-public_likes-free_parking, Query: {'min': 1, 'max': 1000, 'inclusive_min': False, 'inclusive_max': False, 'field': 'reviews.ratings.Overall'}, Status: True, Response: 200, Total Hits: 3991, Result Status: {'total': 2, 'failed': 1, 'successful': 1, 'errors': {'bucket_bucket4_idx_uivgw-Overall-public_likes-free_parking_62aea4fa61346f72_b6d0c5f9': 'grpc_client: query got status code: 503, resp: (*bleve.SearchResult)(nil), err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.106.134:19130: connect: connection refused"'}}
       
       
       

       

      Snippet from logs:

      172.23.97.151 : fts
      2021-03-10T17:59:31.029-08:00 [ERRO] grpc_client: search err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.106.134:19130: connect: connection refused" -- cbft.(*GrpcClient).SearchRPC() at grpc_client.go:254
      2021-03-10T17:59:31.037-08:00 [ERRO] grpc_client: search err: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.23.106.134:19130: connect: connection refused" -- cbft.(*GrpcClient).SearchRPC() at grpc_client.go:254 

      And this time test was at the step of rebalancing out eventing node. Looks like this is nothing to do with above failure.

       [2021-03-10T17:54:15-08:00, sequoiatools/couchbase-cli:7.0:8c597f] rebalance -c 172.23.97.74:8091 --server-remove 172.23.123.31 -u Administrator -p password
      [2021-03-10T17:59:38-08:00, sequoiatools/cmd:44c5b7] 60

       

      Logs:

       

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.106.134.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.120.58.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.120.73.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.120.74.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.120.77.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.120.81.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.120.86.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.121.77.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.123.24.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.123.25.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.123.26.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.123.32.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.96.122.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.96.243.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.96.254.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.105.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.110.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.112.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.148.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.149.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.150.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.151.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.241.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615430952/collectinfo-2021-03-11T024913-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty