Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7438

Couchbase can't use RAM allocated for system disk cache.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 3.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Environment:
    • Triage:
      Untriaged

      Description

      Couchbase will start evicting documents when server memory filled with system disk cache, even when couchbase memory usage is well below bucket quota.
      To prevent this situation most people dropping system disk cache in cron task like this:
      echo 3 > /proc/sys/vm/drop_caches

      Also i don't understand why disk cache created, because Couchbase have no reads from disk(only writes if it set to permanent, like at my servers).

      Discussion in Couchbase Google group: https://groups.google.com/forum/#!topic/couchbase/H52nP3TualU/discussion

      Screenshots of Web UI stats:
      http://gyazo.com/acd03396c42784069cba962dedfe2e6a
      http://gyazo.com/be4897fadfa5fa1c90cac550591e1d39
      http://gyazo.com/3900dfeffc90c2e42f5ce7b7d0fd0f46
      http://gyazo.com/c434b8ead2a7207df9e6a6936ce632ac
      http://gyazo.com/63dd9b44d8e2fe7812ea423a1a748276
      http://gyazo.com/e66503094d3285d95f7a40cc4d8c83fd

      Other data on screenshots is the system disk cache created by Couchbase operations for one day.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        mikew Mike Wiederhold added a comment -

        Andrey,

        Couchbase uses a concept of high and low watermarks in order to evict data. These are set at 75% and 60% of the bucket quota. This means that in your case with a 10GB bucket you will see Couchbase begin to evict documents to disk once the memory quota reaches about 7.5GB and items will continue to be evicted until the memory quota is about 6GB. We also don't allow your cache size to be larger than 90% of the bucket quota since we leave 10% of the memory for management overhead. Does this information explain the behavior your seeing? Please comment on this since I want to try to resolve this issue for our 2.0.2 release.

        Show
        mikew Mike Wiederhold added a comment - Andrey, Couchbase uses a concept of high and low watermarks in order to evict data. These are set at 75% and 60% of the bucket quota. This means that in your case with a 10GB bucket you will see Couchbase begin to evict documents to disk once the memory quota reaches about 7.5GB and items will continue to be evicted until the memory quota is about 6GB. We also don't allow your cache size to be larger than 90% of the bucket quota since we leave 10% of the memory for management overhead. Does this information explain the behavior your seeing? Please comment on this since I want to try to resolve this issue for our 2.0.2 release.
        Hide
        creotiv Andrey Nikishaev added a comment -

        i know this, but the problem that i had less then 60% memory used when couchbase started evicting data to disk, and when i add more memory and lower to 40% used memory it didn't return data to cache. From that time i have 86.5% resident and it not changing.
        When couchbase started to evicting data, i check the system and saw that mostly all memory was used for disk cache, so i think there can be a problem. Also i know that only i using clearing disk cache to fix this problem.

        Show
        creotiv Andrey Nikishaev added a comment - i know this, but the problem that i had less then 60% memory used when couchbase started evicting data to disk, and when i add more memory and lower to 40% used memory it didn't return data to cache. From that time i have 86.5% resident and it not changing. When couchbase started to evicting data, i check the system and saw that mostly all memory was used for disk cache, so i think there can be a problem. Also i know that only i using clearing disk cache to fix this problem.
        Hide
        mikew Mike Wiederhold added a comment -

        Just to be clear, when you say "evicting" do you mean that pages allocated to memcached are swapped to disk?

        Show
        mikew Mike Wiederhold added a comment - Just to be clear, when you say "evicting" do you mean that pages allocated to memcached are swapped to disk?
        Hide
        creotiv Andrey Nikishaev added a comment -

        Yes, they swapped to disk and removed from memcached.

        Show
        creotiv Andrey Nikishaev added a comment - Yes, they swapped to disk and removed from memcached.
        Hide
        mikew Mike Wiederhold added a comment -

        Okay. I will try to run some tests tomorrow to simulate this. Thanks for the quick follow up.

        Show
        mikew Mike Wiederhold added a comment - Okay. I will try to run some tests tomorrow to simulate this. Thanks for the quick follow up.
        Hide
        mikew Mike Wiederhold added a comment -

        This can be closed once we finish UPR.

        Show
        mikew Mike Wiederhold added a comment - This can be closed once we finish UPR.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Mike,

        With UPR in now, can this be closed?

        Show
        maria Maria McDuff (Inactive) added a comment - Mike, With UPR in now, can this be closed?
        Hide
        mikew Mike Wiederhold added a comment -

        This should be resolved in 3.0. Please reopen if this issue is still seen.

        Show
        mikew Mike Wiederhold added a comment - This should be resolved in 3.0. Please reopen if this issue is still seen.
        Hide
        thuan Thuan Nguyen added a comment -

        I close this bug since in system test in 3.0.0, we fixed and verified some issues relate to mem use, high and low water mark.

        Show
        thuan Thuan Nguyen added a comment - I close this bug since in system test in 3.0.0, we fixed and verified some issues relate to mem use, high and low water mark.

          People

          • Assignee:
            thuan Thuan Nguyen
            Reporter:
            creotiv Andrey Nikishaev
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes