Description
Add a sub-section to https://docs.couchbase.com/server/current/n1ql/n1ql-manage/monitoring-n1ql-query.html#sys-completed-config something along the lines of:
Save completed requests to local files
To save completed requests to per-node files in the Couchbase server logs directory, set the {{completed-stream-size }}configuration parameter to the size of the data to retain (per-node), in MiB.
Set it to 0 to disable saving of completed requests to files.
The configuration (qualifiers) for system:completed_requests controls which requests are to be saved.
NOTE: The additional processing required to save completed requests to disk may limit overall request throughput on a Query node, but typically only when the configured qualifiers have every request being recorded and requests are very small / short-lived. The speed of the file system on which the server logs directory resides naturally affects the potential impact too.
When configured, completed requests will be written to files with the prefix "local_request_log." in the server logs directory. They are GZIP archives containing JSON entries for each recorded completed request.
Tip: gzip -qdc <file>|jq . will allow easy examination of the file contents.
Multiple entries are included in each file with the intention that the uncompressed size shouldn't exceed 100 MiB, though this is not a hard limit (entries are not truncated to adhere to it). Files may also be written with less content if nothing has been recorded in them for an extended period.
Multiple files are written in parallel so whilst the order of requests in a file is sequential, they are not sequential for the node across all files. Files that are actively being written to are not available for reading and don't count towards the configured size limit until they're finalised.
The server aims to manage and retain files such that the total disk space used is within the configured limit on the node. Older files are removed as necessary to make space for newly finalised files. Given the fact that files are written to in parallel, it isn't guaranteed that only the oldest requests are evicted when a file is removed.
The system:completed_requests_history keyspace is provided for SQL++ access to the archived files, but owing to their nature (external GZIP archives) performance is restricted, particularly with large histories on clusters with multiple Query service nodes. Directly reading the files may be more useful in some cases.
Attachments
Issue Links
- is caused by
-
MB-62492 Provide a configurable option to stream all completed_requests to disk.
-
- Resolved
-