Runbook for validating indexer node is not undersized

Description

We should cover how to check if indexer is undersized in run book.   For this we need to know if indexes are 0 resident ratio because memory quota is too low or index not being used at all. 

From slack
Hi Team,
From a customer's cluster, I'm observing that the resident ratio of the indexes is 0%, does it indicate the cluster is undersized?
 
you can check in indexer.log for these warnings Warning: not enough memory to hold records in memory  which we see in general if memory quota is low. You can also check if there is a possibility of a restart.
 
you can also check the index node level stat in indexer_stats.log avg_resident_percent . This gives a better picture from sizing perspective than looking at the individual index level stat. If an index is not being used or indexer restarts, index resident ratio can be low. You can also double check it with total_data_size node level stat and get a rough idea how much memory would be required at 10% RR.
 
Also, the effective memory quota will be reported as aggregated stat to prometheus starting 7.6 - storage_stats_manager_enterprise.go
 
 

Components

Affects versions

None

Fix versions

Labels

Environment

None

Release Notes Description

None

Activity

Show:
Duplicate
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created October 25, 2023 at 3:28 PM
Updated October 25, 2023 at 4:52 PM
Resolved October 25, 2023 at 4:52 PM
Instabug