Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60847

OOM crashes on FTS after rebalance operation

    XMLWordPrintable

Details

    • Bug
    • Resolution: Known Error
    • Critical
    • None
    • 7.6.0
    • fts
    • Capella
      Couchbase server
      Enterprise Edition 7.6.0 build 2149

    Description

      1. Created a capella cluster with the following configuration

      Service Number of nodes CPU cores Memory Disk size
      KV 3 8 32GB 250GB
      FTS 5 16 32GB 250B
      N1QL 2 4 16GB 250GB

      2. Loaded 1M documents into a bucket. The 'vector_data' has dimension 1536.

      3. Created an index on the vector data.

      4. Wait for indexing to complete.

      5. Started an incremental load where we are inserting new documents as well as updating existing documents.

      6. Loaded new docs till total doc count was 3M. Wait for indexing to complete

      7. Now started running queries on the index.

      8. Started a rebalance out operation(added a new FTS node)

      9. Started a rebalance in operation(removed a FTS node).

      10. After some time I see crashes on FTS

      2024-02-18T17:08:50.877Z, ns_log:0:info:message(ns_1@svc-s-node-006.8tpall0wqm2hzgux.sandbox.nonprod-project-avengers.com) - Service 'fts' exited with status 137. Restarting. Messages:    "total_queries_max_result_window_exceeded_error": 2,    "total_queries_partial_results_error": 0,    "total_queries_rejected_by_herder": 0,    "total_queries_search_in_context_error": 0,    "total_queries_slow": 0,    "total_queries_timeout": 0,    "total_request_time": 128296584378} 

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mohsin.ahmed Mohsin Ahmed
            mohsin.ahmed Mohsin Ahmed
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty