Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51024

Increase the default value of magma_mem_quota_ratio

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 7.1.0
    • 7.1.0
    • ns_server
    • 1
    • Yes

    Description

      The default value of magma_mem_quota_ratio was raised from 10% to 50% here as part of MB-49489. ns_server seems to set the config to 10% as default here. Could we raise it to 50%?

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          dfinlay Dave Finlay added a comment -

          Hi Apaar - this is a big change to make at such a late stage. Is everyone agreed on this? (Perhaps I missed the discussions.)

          dfinlay Dave Finlay added a comment - Hi Apaar - this is a big change to make at such a late stage. Is everyone agreed on this? (Perhaps I missed the discussions.)
          apaar.gupta Apaar Gupta added a comment -

          We discussed this with the kv team when the config was updated in kv_engine's config.json. These changes are part of an MB where we added bloom filter eviction in magma, this was needed cos at 1% dgm + very small item sizes(sub 50 bytes) it was possible for the bloom filters which consume 1 bytes per key to consume the entire bucket quota. The changes evict the bloom filters only if we are at risk of consuming a large part of the bucket quota.

          Magma's internal quota is essentially the same cos we have added magma_mem_quota_low_watermark_ratio which is 20% of magma's memory quota so effectively the same 10% it was before. The remaining 40% of bucket quota(80% of magma quota) is for the bloom filters but will not be used unless we are at very low dgm with a large number of small items.

          apaar.gupta Apaar Gupta added a comment - We discussed this with the kv team when the config was updated in kv_engine's config.json. These changes are part of an MB where we added bloom filter eviction in magma, this was needed cos at 1% dgm + very small item sizes(sub 50 bytes) it was possible for the bloom filters which consume 1 bytes per key to consume the entire bucket quota. The changes evict the bloom filters only if we are at risk of consuming a large part of the bucket quota. Magma's internal quota is essentially the same cos we have added magma_mem_quota_low_watermark_ratio which is 20% of magma's memory quota so effectively the same 10% it was before. The remaining 40% of bucket quota(80% of magma quota) is for the bloom filters but will not be used unless we are at very low dgm with a large number of small items.
          sarath Sarath Lakshman added a comment - - edited

          This change is required to redefine the storage_memory_quota to include the extra bloom filter memory usage which may have been accounted under kv_engine quota when it exceeded above earlier defined magma specific quota of 10%.

          With this change, storage_memory_quota will be the upper limit for magma. Magma will derive a minimum usage quota internally based on this quota, which will be the same as the earlier 10% of the bucket quota. So, this change is to retain the same min memory limit for magma as before. But redefining the storage quota definition. This change was discussed with kv_engine team and the corresponding kv_engine changes have been merged.

          sarath Sarath Lakshman added a comment - - edited This change is required to redefine the storage_memory_quota to include the extra bloom filter memory usage which may have been accounted under kv_engine quota when it exceeded above earlier defined magma specific quota of 10%. With this change, storage_memory_quota will be the upper limit for magma. Magma will derive a minimum usage quota internally based on this quota, which will be the same as the earlier 10% of the bucket quota. So, this change is to retain the same min memory limit for magma as before. But redefining the storage quota definition. This change was discussed with kv_engine team and the corresponding kv_engine changes have been merged.
          dfinlay Dave Finlay added a comment -

          Got it - thanks Apaar and Sarath for the clarification. Sounds good.

          dfinlay Dave Finlay added a comment - Got it - thanks Apaar and Sarath for the clarification. Sounds good.
          drigby Dave Rigby added a comment -

          To put it another way - prior to the changes for MB-49489, Magma wasn't actually obeying the quota which it was assigned anyway...

          drigby Dave Rigby added a comment - To put it another way - prior to the changes for MB-49489 , Magma wasn't actually obeying the quota which it was assigned anyway...

          Build couchbase-server-7.1.0-2333 contains ns_server commit cc01eb7 with commit message:
          MB-51024: Increase magma_mem_quota_ratio from 10% to 50%

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2333 contains ns_server commit cc01eb7 with commit message: MB-51024 : Increase magma_mem_quota_ratio from 10% to 50%

          Build couchbase-server-7.1.0-2348 contains ns_server commit b98ef65 with commit message:
          MB-51024: Fix default value of storage_quota_percentage

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2348 contains ns_server commit b98ef65 with commit message: MB-51024 : Fix default value of storage_quota_percentage

          Verified on 7.1.0-2378

          /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password all -a | grep "magma_mem_quota_ratio"
           ep_magma_mem_quota_ratio:                                  0.5
          [root@kvm-s73105 ~]# curl -s -u Administrator:password localhost:8091/pools/default/buckets/default | jq | grep "storageQuotaPercentage"
            "storageQuotaPercentage": 50,
          

          ankush.sharma Ankush Sharma added a comment - Verified on 7.1.0-2378 /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password all -a | grep "magma_mem_quota_ratio" ep_magma_mem_quota_ratio: 0.5 [root@kvm-s73105 ~]# curl -s -u Administrator:password localhost:8091/pools/default/buckets/default | jq | grep "storageQuotaPercentage" "storageQuotaPercentage": 50,

          People

            ankush.sharma Ankush Sharma
            apaar.gupta Apaar Gupta
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty