Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50166

[CBSE] Further improvement to minMemory calculations in planner for Plasma indexes

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      When planner is doing minMemory calculations there are two cases which need to be studied further  - applicable only for Plasma

      case 1) Index with "0" percent resident ratio they are approximated to use memory based on datasize but even with 0% resident ratio these index have fixed memory usage for skip list etc which needs to be accounted for. This can be seen in following experiment

      two equivalent indexes (no mutations, no  scan in progress) first index with some 20 - 50 % resident ratio (basically any number greater than 0%), second index with 0% resident ratio. In this case minMemory calculation for second index is very low and not linearly proportional with first index. (infact first index which has 20-50% resident ratio will show much higher minMemory requirements)

      case 2) We apply resident_percent to totalMemory used by an index 

      totalMemory = indexMemory + IndexOverhead, here overhead is again a function of indexMemory. But index memory has two parts - one is fixed cost other is variable

      indexMemory = mem_size + mem_size_index, here mem_size_index is the fixed part and represents the cost of datastructs like skiplist, so even with resident ratio 0 we still have mem_size_index in memory, mem_size is memory used for index pages which varies based on resident ratio.

      mem_size_index is usually smaller portion compared to mem_size and with higher resident ratios it does not matter much in overall computation. But when resident percent is very low the mem_size_index's weight in calculations goes up and that creates variance of 10-15% in min memory calculation. This leads to calculated minMemory to be more than what it should be by at least 10-15%. (10 to 15% here is a anecdotal number based on my limited experiments) 

      Due to this we need to base the overhead calculation as well as min memory calculation on only the variable part (i.e on the mem_size stat)

      Attachments

        For Gerrit Dashboard: MB-50166
        # Subject Branch Project Status CR V

        Activity

          People

            shivansh.rustagi Shivansh Rustagi
            yogendra.acharya Yogendra Acharya (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There is 1 open Gerrit change

                PagerDuty