Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.6.2
-
Untriaged
-
0
-
Unknown
Description
Issue:
FTS process crashed with exist status 130. Seems like this happened during jemalloc calls `arena 0 background thread creation faile`
Cluster config:
- 6 search nodes, 3 data nodes and 2 query nodes
- Search service memory quota - 16851
Test config
- Having 2 collections with continuous expiry of the document for every 3600 seconds
- Above 2 collections will also have continuous dataload every 3600 seconds
- Other 3 collections have 10M documents each
- We have 7 indices, but only two indexes are active with continuous data load on the collections along with queries running.
Service 'fts' exited with status 130. Restarting. Messages: |
<jemalloc>: arena 0 background thread creation failed (11) |
OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 8: Resource temporarily unavailable |
OpenBLAS blas_thread_init: RLIMIT_NPROC 95782 current, 95782 max |
Logs:
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.105.122.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.106.171.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.106.176.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.106.30.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.96.230.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.96.245.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.97.100.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.97.108.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.97.109.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.97.66.zip
- https://cb-engineering.s3.amazonaws.com/ashok-on-prem-system-test-oom/collectinfo-2024-05-26T002707-ns_1%40172.23.97.67.zip