[Guardrails] Blocking of KV mutations differs based on the Ops Rate used for data loading

Steps:

1. Initialise a 2 node Couchbase cluster running 'kv' service.
2. Create a CouchStore bucket with 100MB RAM Quota per node. The bucket has 3 scopes, 3 collections and no replicas.
3. Set couchstoreMinimum=50 in this endpoint http://172.23.106.242:8091/settings/resourceManagement

{"bucket":{"collectionsPerQuota":{"enabled":true,"maximum":1},"dataSizePerNode":{"couchstoreMaximum":1.6,"enabled":true,"magmaMaximum":16},"residentRatio":{"couchstoreMinimum":50,"enabled":true,"magmaMinimum":1}},"coresPerBucket":{"enabled":true,"minimum":0.4},"diskUsage":{"enabled":true,"maximum":85}} 

# First, start data loading (creates) with an ops rate of 10,000 ops/sec. The document size is 1024 bytes (1KB). Load until we see an exception 'BUCKET_RESIDENT_RATIO_TOO_LOW'.

1. The number of items in the cluster when the KV mutations were blocked = 597,000.
   Resident ratio shown on the UI = 20.8%
   Using the below mentioned formula, RR = 19.23%

kv_ep_max_size / on(bucket) (sum by(bucket, name) (kv_logical_data_size_bytes{state="active"}))

The metrics in the above formula were fetched from the following endpoints,
http://172.23.107.110:8091/_prometheus/api/v1/query?query=kv_ep_max_size
http://172.23.107.110:8091/_prometheus/api/v1/query?query=kv_logical_data_size_bytes{state=%22active%22}

1. The same test was executed (create a bucket, load data until we see the exception 'BUCKET_RESIDENT_RATIO_TOO_LOW'), but this time, it is tried with an ops rate of 100,000 ops/sec (instead of 10,000 ops/sec in the previous scenario) and the following was observed.
2. The number of items in the cluster when the KV mutations were blocked = 1,743,556
   Resident ratio shown on the UI = 7.57%
   Using the above mentioned formula, RR = 6.59%

The two concerns that I see are:
KV mutations are not blocked when the configured minimum resident ratio is reached.
Blocking of KV mutations depends on the ops rate and hence, we see a difference in the resident ratio and the number of items loaded.