Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58593

[Guardrails] KV mutations are not stopped at the configured minimum resident ratio guardrail

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 7.6.0
    • couchbase-bucket, ns_server
    • Couchbase server version = 7.6.0-1460
      OS = Debian 10
    • Untriaged
    • Linux x86_64
    • 0
    • Unknown

    Description

      Steps to repro:

      1. Initialise a 2 node cluster running just the 'kv' service.
      2. Create a CouchStore bucket with RAM Quota 100MB per node. The bucket has 3 scopes and 3 collections.
      3. Configure couchstoreMinimum=50 in this rest endpoint http://172.23.106.242:8091/settings/resourceManagement

      {"bucket":{"collectionsPerQuota":{"enabled":true,"maximum":1},"dataSizePerNode":{"couchstoreMaximum":1.6,"enabled":true,"magmaMaximum":16},"residentRatio":{"couchstoreMinimum":50,"enabled":true,"magmaMinimum":1}},"coresPerBucket":{"enabled":true,"minimum":0.4},"diskUsage":{"enabled":true,"maximum":85}}

      1. Start data loading (creates) with document size = 1KB (1024 bytes).
      2. Load until we get an exception saying 'BUCKET_RESIDENT_RATIO_TOO_LOW'.
      3. Calculate resident ratio at this point, using the formula,

      kv_ep_max_size / on(bucket) (sum by(bucket, name) (kv_logical_data_size_bytes{state="active"}))

      1. Resident ratio = 104857600 / (442972497 + 446644811) = 11.78%

      The above values were fetched from these endpoints,

      http://172.23.107.110:8091/_prometheus/api/v1/query?query=kv_ep_max_size

      {"status":"success","data":{"resultType":"vector","result":[{"metric":
      		 
      {"__name__":"kv_ep_max_size","bucket":"default","instance":"kv","job":"kv_high_cardinality","name":"kv_ep_max_size"}
      		 
      ,"value":[1694422292.280,"104857600"]}]}}

       

      http://172.23.106.242:8091/_prometheus/api/v1/query?query=kv_logical_data_size_bytes{state=%22active%22}

      {"status":"success","data":{"resultType":"vector","result":[{"metric":
      		 
      {"__name__":"kv_logical_data_size_bytes","bucket":"default","instance":"kv","job":"general","name":"kv_logical_data_size_bytes","state":"active"}
      		 
      ,"value":[1694422571.122,"446644811"]}]}}

       

      http://172.23.107.110:8091/_prometheus/api/v1/query?query=kv_logical_data_size_bytes{state=%22active%22}

      {"status":"success","data":{"resultType":"vector","result":[{"metric":
      		 
      {"__name__":"kv_logical_data_size_bytes","bucket":"default","instance":"kv","job":"general","name":"kv_logical_data_size_bytes","state":"active"}
      		 
      ,"value":[1694422599.353,"442972497"]}]}}

      Resident ratio shown on the UI was 13%.

      Since we configured the minimum resident ratio for a CouchStore bucket to be 50%, KV mutations shouldn't be allowed beyond that.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-58595 [Guardrails] Blocking of KV mutations differs based on the Ops Rate used for data loading

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vibhav.sp Vibhav S P
              vibhav.sp Vibhav S P
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty