Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55106

[Serverless]: FTS is are oom killed and auto-failed over.



    • Bug
    • Resolution: Fixed
    • Critical
    • Elixir
    • Elixir
    • fts
    • 7.5.0-3512


      1. Create 20 serverless databases. wait for kv to scale from 3 to 6 kv nodes.
      2. Load 2M items in all databases.
      3. Create 2 gsi index and 2 fts indexes.
      4. Run 2 GSI and 2 fts QPS on each database.
      5. Repeat the above steps until there are 80 databases in system.
      6. GSI/FTS/KV nodes were kept on adding to the cluster based on number of GSI tenants, FTS CPU/RAM consumption, and number of databases created in each iteration.
      7. At this load FTS nodes in the cluster started auto-failing over due to oom killed:

      Node ns_1@tx01cqx1djqmbtm.r1tpkpve0q4kf7do.sandbox.nonprod-project-avengers.com

      Service 'fts' exited with status 137. Restarting. Messages:
      2023-01-14T07:33:49.861+00:00 [ERRO] rest: error code: 429, msg: rest_index: Query, indexName: volumetestbucket-52-2c6v2t._default.volumetestbucket_52_2c6v2t_fts_idx_0, err: query request rejected -- rest.ShowErrorBody() at rest.go:62
      2023-01-14T07:33:49.937+00:00 [INFO] app_herder: query ended, indexes: 74, waiting: 50
      2023-01-14T07:33:49.956+00:00 [INFO] app_herder: query ended, indexes: 74, waiting: 50
      2023-01-14T07:33:50.066+00:00 DEBU REGU.throttle.(*throttleimpl).sendServiceAggregatedReports() at servicereport.go:242 [id 201] setting /regulator/report/t/fts/59db338847ab6227d1a1f1d30d53cc14 to [01d8e6cbf9da301a766f...3464746a003400000100] (40 bytes)
      2023-01-14T07:33:50.890+00:00 [INFO] app_herder: merger progress, indexes: 74, waiting: 50
      2023-01-14T07:33:51.341+00:00 [INFO] app_herder: indexing over indexQuota: 19660800000, memUsed: 27211297525, preIndexingMemory: 28427229, indexes: 74, waiting: 50
      2023-01-14T07:34:01.986+00:00 [WARN] (GOCBCORE) Config block decode failure (context canceled) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:689
      2023-01-14T07:34:00.467+00:00 [WARN] (GOCBCORE) Config block decode failure (context canceled) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:689


        No reviews matched the request. Check your Options in the drop-down menu of this sections header.



            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            0 Vote for this issue
            4 Start watching this issue



              Gerrit Reviews

                There are no open Gerrit changes