Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62057

Vector Search: FTS crash looping with exit status 1 on system-test cluster

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Build:
      *7.6.2-3674*
      Issue:

      FTS process continuously crashing with exist status 1. Based on initial analysis, this may be happening during query.

      Cluster config:

      • 6 search nodes, 3 data nodes and 2 query nodes
      • Search service memory quota - 16851

      Test config

      • Having 2 collections with continuous expiry of the document for every 3600 seconds
      • Above 2 collections will also have continuous dataload every 3600 seconds
      • Other 3 collections have 10M documents each
      • We have 7 indices, but only two indexes are active with continuous data load on the collections along with queries running.
      •  

      Service 'fts' exited with status 1. Restarting. Messages:
      		 
      2024-05-25T15:14:29.469-07:00 [WARN] grpc_client: Query() returned error from host: 172.23.106.171:9130, err: grpc_client: query got status code: 429, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc0be39ca80), Request:(*bleve.SearchRequest)(0xc0b23fa400), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = ResourceExhausted desc = grpc_server: Search query reject on not enough quota: query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:153
      		 
      2024-05-25T15:14:30.170-07:00 [WARN] grpc_client: Query() returned error from host: 172.23.106.171:9130, err: grpc_client: query got status code: 429, resp: &bleve.SearchResult{Status:(*bleve.SearchStatus)(0xc0bd4450e0), Request:(*bleve.SearchRequest)(0xc0b14c4300), Hits:search.DocumentMatchCollection(nil), Total:0x0, Cost:0x0, MaxScore:0, Took:0, Facets:search.FacetResults(nil)}, err: rpc error: code = ResourceExhausted desc = grpc_server: Search query reject on not enough quota: query request rejected -- cbft.(*GrpcClient).SearchInContext.func1() at grpc_client.go:153
      		 
       
      		 
      libgomp: Thread creation failed: Resource temporarily unavailable
      		 
       
      		 
      libgomp: Thread creation failed: Resource temporarily unavailable

      Logs:

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-62051 FTS OOM - Service 'fts' exited with status 137

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. MB-62058 Vector Search: FTS crashed with exit status 130

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              aditi.ahuja Aditi Ahuja
              ashokkumar.alluri Ashok Alluri
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty