Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.1.0
-
Untriaged
-
1
-
Unknown
Description
When planner is doing minMemory calculations there are two cases which need to be studied further - applicable only for Plasma
case 1) Index with "0" percent resident ratio they are approximated to use memory based on datasize but even with 0% resident ratio these index have fixed memory usage for skip list etc which needs to be accounted for. This can be seen in following experiment
two equivalent indexes (no mutations, no scan in progress) first index with some 20 - 50 % resident ratio (basically any number greater than 0%), second index with 0% resident ratio. In this case minMemory calculation for second index is very low and not linearly proportional with first index. (infact first index which has 20-50% resident ratio will show much higher minMemory requirements)
case 2) We apply resident_percent to totalMemory used by an index
totalMemory = indexMemory + IndexOverhead, here overhead is again a function of indexMemory. But index memory has two parts - one is fixed cost other is variable
indexMemory = mem_size + mem_size_index, here mem_size_index is the fixed part and represents the cost of datastructs like skiplist, so even with resident ratio 0 we still have mem_size_index in memory, mem_size is memory used for index pages which varies based on resident ratio.
mem_size_index is usually smaller portion compared to mem_size and with higher resident ratios it does not matter much in overall computation. But when resident percent is very low the mem_size_index's weight in calculations goes up and that creates variance of 10-15% in min memory calculation. This leads to calculated minMemory to be more than what it should be by at least 10-15%. (10 to 15% here is a anecdotal number based on my limited experiments)
Due to this we need to base the overhead calculation as well as min memory calculation on only the variable part (i.e on the mem_size stat)
Attachments
Gerrit Reviews
For Gerrit Dashboard: MB-50166 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
206263,3 | MB-50166 Adding MemSizeIndex stat to planner filtered stats map | unstable | indexing | Status: NEW | 0 | 0 |