Structured Logs

Background

As the complexity and scale of applications grow, the need for detailed, structured, and easily parsable logs becomes paramount. Structured Logging provides a consistent format for logs, making it easier to analyze, monitor, and troubleshoot issues in real-time. We currently expose our debugging/support logs to customers, which are constantly modified and do not provide any formatting and backwards compatibility guarantees.

Structured Logs

Structured logging is the practice of implementing a consistent, predetermined message format for application logs that allows them to be treated as data sets that can be more easily searched and analyzed than text [1]. Structured logs are often structured in a format like JSON or XML and usually contain well-defined and contextual information needed for diagnosis, including error messages and other properties like event number, hostname, timestamp, and request [2]. 

Use Cases

Real-time Monitoring: Mobile Developers and Ops teams can monitor Sync Gateway operations in real-time. Structured logs allow for easier integration with monitoring tools and dashboards, providing a clear view of data flow and potential bottlenecks. They also facilitate formatting guarantees which persist across versions and provide higher levels of backwards compatibility.

Integration with Third-Party Providers: Structured Logs can be easily integrated with popular log management solutions like Datadog, Sumologic, Splunk, or New Relic. This allows for advanced log analysis, visualization, and alerting. Structured Logs help customers transform the logs output into what fits their observability use cases in a more maintainable way across different types of logs and Sync Gateway releases.

Troubleshooting and Debugging: When issues arise, developers need to pinpoint the root cause quickly. Structured Logs provide detailed insights into the SG operations, enabling faster identification of issues and reducing downtime.

App Services Output

{

   "date":1697061004.477,

   "log":"2023-10-11T21:50:04.477Z [INF] c:#29682 db:asset360 \tBulkDocs: Doc \"<ud>ec-114344450-cee7-05042023-2</ud>\" --> 403 Document rejected  (403 Document rejected )",

   "timestamp":"2023-10-11T21:50:04.477Z",

   "app_endpoint":"asset360",

   "capella_app_service_version":"3.1.2-0.0.1",

   "capella_app_service_id":"79f7a173-0825-4994-8212-b35d90833be5",

   "capella_app_service_node_id":"79f7a173-0825-4994-8212-b35d90833be5:i-0720dd33c5178ec2b",

   "capella_cluster_id":"ade5cc3b-c6b5-4aed-a90f-36ef7e722c87",

   "capella_tenant_id":"13fea298-2f3b-429a-a645-4e1c3b2f45ae",

   "level":"info"

} 

Output

Requirements

• Format consistency: Logs must have a consistent format (key-value) that is applicable to Sync Gateway Logs. 
• Transformation of existing logs: All existing logs must adhere to that format. 
• Future proof logs (validation): Log formatting must be maintained as new logs are added to Sync Gateway or existing logs are modified.
• Support two separate modes of logs formatting and consumption: support logs and user facing (structured) logs
• Required fields/schema:
  • Various IDs such as: "appsvc_id", "node_id", "cluster_id", "tenant_id"
  • HTTP Status code: 2xx, 3xx, 4xx
  •  

References:

1) [SumoLogic Structured Logging|https://www.sumologic.com/glossary/structured-logging/]

2) SemaText Structured Logs