Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Untriaged
-
1
-
Unknown
Description
There is a OOM when running 10K indexes initial build test. The test seems to run fine on 7.1.0-1085 and 7.0.2-6558. The issue is due to large amount of buf memory (over 200G). The number of sctxs is 327K. Buf memory per sctxs is about 611K.
2021-08-24T17:02:35.479-07:00 [Info] Periodic Aggregated StorageStats:
|
{
|
"num_shard": 200,
|
"num_instances": 20000,
|
"num_lssctx": 400,
|
"num_sctxs": 328980,
|
"num_active_sctxs": 1605,
|
"num_free_sctxs": 327291,
|
"num_wctxs": 222167,
|
"num_free_wctxs": 202167,
|
"num_active_wctxs": 20000,
|
"num_readers": 0,
|
"num_writers": 113268,
|
"cpu": 47.42503,
|
"total_core": 48,
|
"threads": 7581,
|
"assigned_quota": 165150720000,
|
"current_quota": 33030144000,
|
"memory_resident": 225041768448,
|
"memory_jemalloc": 274305875968,
|
"memory_jemalloc_allocated": 264499729248,
|
"memory_in_use": 274305474079,
|
"memory_stats_size_total": 234419002140,
|
"memory_stats_size_global": 32935743620,
|
"memory_stats_size_page": 32772875152,
|
"memory_stats_size_index": 513366504,
|
"memory_stats_size_bloom": 0,
|
"memory_stats_size_delta": 1914167456,
|
"reclaim_pending": 77305974,
|
"buf_memused": 201055454510,
|
Looking at the same run with 7.1.0-1085, the number of sctxs is about 71K with buf memory using around 40G. Buf memory per sctx is about 540K. But the log in this run is rotated, but indexer RSS is stable for HTML report. There is also no crash.
2021-07-25T10:15:50.145-07:00 [Info] Periodic Aggregated StorageStats:
|
{
|
"num_shard": 200,
|
"num_instances": 20000,
|
"num_lssctx": 400,
|
"num_sctxs": 71350,
|
"num_active_sctxs": 3834,
|
"num_free_sctxs": 69408,
|
"num_wctxs": 164843,
|
"num_free_wctxs": 142515,
|
"num_active_wctxs": 22171,
|
"num_readers": 0,
|
"num_writers": 75218,
|
"cpu": 26.52322,
|
"total_core": 48,
|
"threads": 1750,
|
"assigned_quota": 165150720000,
|
"current_quota": 165150720000,
|
"memory_resident": 172858757120,
|
"memory_jemalloc": 162312081408,
|
"memory_jemalloc_allocated": 159128242768,
|
"memory_in_use": 162312073216,
|
"memory_stats_size_total": 146420139857,
|
"memory_stats_size_global": 106864972696,
|
"memory_stats_size_page": 106694784590,
|
"memory_stats_size_index": 417930187,
|
"memory_stats_size_bloom": 0,
|
"memory_stats_size_delta": 2787156936,
|
"reclaim_pending": 258622480,
|
"buf_memused": 39048802600,
|
We need to find out why there is a huge jump in number of sctx.