Couchbase Server should have warnings for high system memory usage

Sometimes memory use by Couchbase Server processes can rise slowly over time due to a number of factors (workload, accumulation of stats, bugs). The server is already measuring memory use and if it sees memory use cross thresholds it should fire an escalating series of alerts.

The following are suggested:

• if memory used > 85% of total memory fire alert:
  • "Notice: system memory use is {mem_used_percent} of total available memory, above the notice threshold of {notice_percent}"
• if memory used > 90% of total memory fire alert:
  • "Warning: system memory use is {mem_used_percent} of total available memory, above the warning threshold of {notice_percent}"
• if memory used > 95% of total memory fire alert:
  • "CRITICAL: system memory use is {mem_used_percent} of total available memory, above the critical threshold of {notice_percent}"