Queries get constantly rejected by app herder due to high idle memory usage

Steps to reproduce

1. Created a 1 node cluster on MacOS 14 Sonoma x86(on a EC2 Mac Mini)

2. Created a magma bucket named "test_bucket" and loaded 10k documents with vectors of 1536 dims

3. Created a vector index - definition attached

4. Post indexing completion ran a create workload and number of docs increased from 10k to around 35k. The ops rate was around 100-200 ops/sec

5. Stopped the query workload at around 35k docs and started an update (mutation) workload. The ops rate was around 100-200 ops/sec

6. Continuously ran a query workload during step 4 and 5. The rate was around 40-60 qps

7. Stopped the query workload and kv workload and waited for 30 minutes

8. It can be seen that memory usage does not reduce

9. Started the mutation and query workload after 30 minutes. All queries are constantly rejected by app-herder 

2024-04-24T20:05:20.863+00:00 [ERRO] rest: error code: 429, msg: rest_index: Query, indexName: test_bucket._default.test_index, err: query request rejected -- rest.ShowErrorBody() at rest.go:59

2024-04-24T20:05:20.882+00:00 [INFO] app_herder: querying over queryQuota: 8388608000, estimated size: 96, runningQueryUsed: 2120, memUsed: 8395212600 

Logs for 2 instances of the same

1. MacOS Sonoma x86 - 
https://cb-engineering.s3.amazonaws.com/Mac_vec_search/collectinfo-2024-04-24T210815-ns_1%40127.0.0.1.zip
2. MacOS Ventura x86 - https://cb-engineering.s3.amazonaws.com/Mac_vec_search/collectinfo-2024-04-24T210828-ns_1%40127.0.0.1.zip
 
The definition of index - https://cb-engineering.s3.amazonaws.com/Mac_vec_search/index_def.json