Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60424

FTS with 75G total ram is OOM killed with just 4 vector indexes(384 dimensions) with 1M items in each during rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.6.0
    • 7.6.0
    • fts
    • 7.6.0-2017

    Description

      1. Create a 2 KV, 4 FTS nodes cluster. FTS service RAM quota is 19037MB
      2. Create 1 Magma bucket with 2 collections.
      3. Start loading fashion data with 384 dimensional embedding data on product description. 1M in each collection.
      4. Create fts vector indexes 2 for each collection. In total 4 FTS indexes
      5. Start an upsert workload on vector fields
      6. Start 10 queries per sec workload randomly running on any FTS index
      7. Rebalance in a FTS node - Passed
      8. Rebalance out a FTS node - Passed
      9. Rebalance in 2 nodes, out 1 FTS node - Passed
      10. Swap rebalance FTS node - Passed
      11. Failover 1 node and RebalanceOut that node - Rebalance failed

      172.23.107.237

      		 
      Rebalance exited with reason {service_rebalance_failed,fts,
      		 
      {agent_died,<35518.29035.1>,
      		 
      {lost_connection,
      		 
      {'ns_1@172.23.107.221',shutdown}}}}.
      		 
      Rebalance Operation Id = a479bfcb5cca4dda9e4644311e1a57b4
      		 
       
      		 
      Service 'fts' exited with status 137. Restarting. Messages:
      		 
      2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 3 for host: b405d5108d6d3bd77a1e9e4ec411c224-172.23.107.237:19130
      		 
      2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 4 for host: b405d5108d6d3bd77a1e9e4ec411c224-172.23.107.237:19130
      		 
      2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 0 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
      		 
      2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 1 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
      		 
      2024-01-16T23:16:30.919-08:00 [INFO] grpc_client: grpc ClientConn Created 2 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
      		 
      2024-01-16T23:16:30.920-08:00 [INFO] grpc_client: grpc ClientConn Created 3 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
      		 
      2024-01-16T23:16:30.920-08:00 [INFO] grpc_client: grpc ClientConn Created 4 for host: 39a447ba1854e950a75b534411c3adf5-172.23.107.232:19130
      		 
      2024-01-16T23:20:24.354-08:00 [WARN] (GOCBCORE) Config block decode failure (context canceled) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:742

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty