Major Description
Query has now an unfair memory quota mechanism (per node) that goes alongside the fair (per request) memory quota.
Document the node_quota /admin/settings parameter as well as the upcoming UI setting.
Per node memory sets the maximum amount of process memory after which the garbage collector will be very aggressive in keeping memory down.
The limit is a soft limit, and we should be very clear about this.
This means that the garbage collector will try very hard to keep memory to the set limit, but there is no guarantee that it may succeed.
A fraction of this memory limit (currently set at 67%) acts as a top limit for the amount of memory documents loaded from the KV can use within the query service.
Other items (such as caches, prepared plans, execution trees and memory used by indexing clients or javascript evaluators) are not tracked.
This is akin to the request quota, but on a per node basis - if the overall document memory exceeds this value, those requests that have tipped memory usage over the edge will start having memory allocation errors until document memory usage node wide goes down.
Default value is 0, which is unlimited on both fronts.
An appropriate value for this setting would be a fraction (90%?) of the machine memory, so as to leave enough memory for other services, other processes and OS memory.