Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4249

Better calculate high/low watermarks for large RAM systems

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.8.1, 2.0, 2.0.1
    • Fix Version/s: feature-backlog
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Triage:
      Untriaged
    • Is this a Regression?:
      Yes

      Description

      Our current method of taking 60% and 75% of RAM for low/high watermarks leaves us with lots of wasted space on systems with larger amounts of RAM (16GB+).

      I would propose a default of 5GB below max for high water mark and 10GB below max for low watermark for any bucket allocation over 16GB. That will make it only really kick in for 32GB systems.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        drigby Dave Rigby added a comment -

        Assigning this back to reporter - Perry Krug is this still an issue, or is this simply a per-customer sizing exercise?

        Do you have any information on the kinds of sizing recommendation SEs currently use for customer deployments of various quota sizes?

        Show
        drigby Dave Rigby added a comment - Assigning this back to reporter - Perry Krug is this still an issue, or is this simply a per-customer sizing exercise? Do you have any information on the kinds of sizing recommendation SEs currently use for customer deployments of various quota sizes?
        Hide
        perry Perry Krug added a comment -

        Hi Dave, I do think this is still an issue. Sizing does take care of how much RAM is needed and it does use these same percentages. However, my point with this issue is more around how much RAM we might be "wasting" by only using percentages.

        For instance, when running Couchbase on a node with 128GB of RAM, we cut off 80% for the server quota which is >20GB. When a bucket is created, it only has 100GB to work with and then we take another 75% of that for the high water mark (another 25GB). I realize there are various inefficiencies with memory fragmentation and that larger amounts of data in RAM may require larger buffers...but I think nearly 50GB of unusable RAM is too much. And it gets even higher with nodes of 256GB and higher.

        We do have reasonable support for reducing that first 80%, but the product doesn't really have a good way for managing high/low watermarks and so I was proposing that we enhance our calculation to take this into account and have fixed values (5GB/10GB?) if the percentages are too high.

        Show
        perry Perry Krug added a comment - Hi Dave, I do think this is still an issue. Sizing does take care of how much RAM is needed and it does use these same percentages. However, my point with this issue is more around how much RAM we might be "wasting" by only using percentages. For instance, when running Couchbase on a node with 128GB of RAM, we cut off 80% for the server quota which is >20GB. When a bucket is created, it only has 100GB to work with and then we take another 75% of that for the high water mark (another 25GB). I realize there are various inefficiencies with memory fragmentation and that larger amounts of data in RAM may require larger buffers...but I think nearly 50GB of unusable RAM is too much. And it gets even higher with nodes of 256GB and higher. We do have reasonable support for reducing that first 80%, but the product doesn't really have a good way for managing high/low watermarks and so I was proposing that we enhance our calculation to take this into account and have fixed values (5GB/10GB?) if the percentages are too high.
        Hide
        drigby Dave Rigby added a comment -

        Thanks for the input Perry.

        I think it makes sense to split this into two items:

        1) to allow much easier setting of the high/low watermarks (i.e. cluster-wide) - currently you have to use cbecptl which doesn't survive a restart.
        2) Some kind of smarter default algorithm along the lines of what you describe.

        I propose this as it's relatively "easy" for us to do (1), as we just expose existing settings, and for users which care (i.e. larger deployments) this probably adds a lot of value without a huge amount of effort.

        For (2) we'd need to re-calibrate the correct values; and additionally it does make it harder to communicate to users how the watermark system operates if it's some combination of fixed minimum + percentage-based value.

        Show
        drigby Dave Rigby added a comment - Thanks for the input Perry. I think it makes sense to split this into two items: 1) to allow much easier setting of the high/low watermarks (i.e. cluster-wide) - currently you have to use cbecptl which doesn't survive a restart. 2) Some kind of smarter default algorithm along the lines of what you describe. I propose this as it's relatively "easy" for us to do (1), as we just expose existing settings, and for users which care (i.e. larger deployments) this probably adds a lot of value without a huge amount of effort. For (2) we'd need to re-calibrate the correct values; and additionally it does make it harder to communicate to users how the watermark system operates if it's some combination of fixed minimum + percentage-based value.
        Hide
        perry Perry Krug added a comment -

        Thanks Dave, I think that's an accurate assessment.

        Personally I would prefer that we do #2 to make Couchbase better "out of the box" for all our users. FWIW, we don't really get into the details of watermark calculation and operation other than a few places in our training material/docs and that vast majority of users don't really see or think about them.

        #2 would be a nice improvement to say "we've made Couchbase more efficient for larger RAM systems"

        Show
        perry Perry Krug added a comment - Thanks Dave, I think that's an accurate assessment. Personally I would prefer that we do #2 to make Couchbase better "out of the box" for all our users. FWIW, we don't really get into the details of watermark calculation and operation other than a few places in our training material/docs and that vast majority of users don't really see or think about them. #2 would be a nice improvement to say "we've made Couchbase more efficient for larger RAM systems"
        Hide
        drigby Dave Rigby added a comment -

        Hi Perry,

        I agree we should ideally do both, it's more a case that (1) is much easier than (2), and hence if we split them (and do 1 first) then we get some earlier incremental benefit than waiting for both.

        Show
        drigby Dave Rigby added a comment - Hi Perry, I agree we should ideally do both, it's more a case that (1) is much easier than (2), and hence if we split them (and do 1 first) then we get some earlier incremental benefit than waiting for both.

          People

          • Assignee:
            drigby Dave Rigby
            Reporter:
            perry Perry Krug
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Gerrit Reviews

              There are no open Gerrit changes