Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Elixir
-
7.5.0-3512
-
Untriaged
-
-
0
-
Unknown
Description
- Create 20 serverless databases. wait for kv to scale from 3 to 6 kv nodes.
- Load 2M items in all databases.
- Create 2 gsi index and 2 fts indexes.
- Run 2 GSI and 2 fts QPS on each database.
- Repeat the above steps until there are 80 databases in system.
- GSI/FTS/KV nodes were kept on adding to the cluster based on number of GSI tenants, FTS CPU/RAM consumption, and number of databases created in each iteration.
- At this load FTS nodes in the cluster started auto-failing over due to oom killed:
|
Node ns_1@tx01cqx1djqmbtm.r1tpkpve0q4kf7do.sandbox.nonprod-project-avengers.com |
Service 'fts' exited with status 137. Restarting. Messages:
|
2023-01-14T07:33:49.861+00:00 [ERRO] rest: error code: 429, msg: rest_index: Query, indexName: volumetestbucket-52-2c6v2t._default.volumetestbucket_52_2c6v2t_fts_idx_0, err: query request rejected -- rest.ShowErrorBody() at rest.go:62
|
2023-01-14T07:33:49.937+00:00 [INFO] app_herder: query ended, indexes: 74, waiting: 50
|
2023-01-14T07:33:49.956+00:00 [INFO] app_herder: query ended, indexes: 74, waiting: 50
|
2023-01-14T07:33:50.066+00:00 DEBU REGU.throttle.(*throttleimpl).sendServiceAggregatedReports() at servicereport.go:242 [id 201] setting /regulator/report/t/fts/59db338847ab6227d1a1f1d30d53cc14 to [01d8e6cbf9da301a766f...3464746a003400000100] (40 bytes)
|
2023-01-14T07:33:50.890+00:00 [INFO] app_herder: merger progress, indexes: 74, waiting: 50
|
2023-01-14T07:33:51.341+00:00 [INFO] app_herder: indexing over indexQuota: 19660800000, memUsed: 27211297525, preIndexingMemory: 28427229, indexes: 74, waiting: 50
|
2023-01-14T07:34:01.986+00:00 [WARN] (GOCBCORE) Config block decode failure (context canceled) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:689
|
2023-01-14T07:34:00.467+00:00 [WARN] (GOCBCORE) Config block decode failure (context canceled) -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:689
|
