Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Unresolved
-
2.1.0
-
Security Level: Public
-
CBG Sprint 12, CBG Sprint 13
-
8
Description
2.1 removed the option to clean up rotated log files based on number of backups (which could be directly related to file size with max_size*max_backups)
Since we now gzip rotated log files, and have 4 separate log files, this calculation breaks down, and we can no longer rely on it to calculate disk usage.
We need to have a way to limit the total size of logs, as a safeguard to prevent a node's disk from being completely filled up.
As a bonus, we could look into removing the less important logs first (debug, info) to preserve errors and warnings.
Migrated from: https://github.com/couchbase/sync_gateway/issues/3713
Attachments
Gerrit Reviews
Activity
| Field | Original Value | New Value |
|---|---|---|
| Rank | Ranked higher |
| Assignee | The One [ the one ] | Ben Brooks [ ben.brooks ] |
| Labels | needs-info |
| Story Points | 8 |
| Sprint | CBG Sprint 12 [ 749 ] |
| Rank | Ranked higher |
| Planned End | 2019-02-11 14:00 (started sprint) | |
| Planned Start | 2019-02-11 14:00 (started sprint) |
Tasks:
- Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level). If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold.
- Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation)
- If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)
| Assignee | Ben Brooks [ ben.brooks ] | Jacques Rascagneres [ jacques.rascagneres ] |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Actual Start | 2019-02-18 05:34 (issue has been started) |
| Status | In Progress [ 3 ] | In Review [ 10107 ] |
Automated transition triggered when Jacques Rascagneres created pull request #3972 in GitHub -
| Status | In Review [ 10107 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | In Review [ 10107 ] |
| Sprint | CBG Sprint 12 [ 749 ] | CBG Sprint 12, CBG Sprint 13 [ 749, 753 ] |
| Planned End | 2019-02-11 14:00 | 2019-02-19 13:30 (started sprint) |
| Planned Start | 2019-02-11 14:00 | 2019-02-19 13:30 (started sprint) |
Automated transition triggered when Adam Fraser created pull request #3977 in GitHub -
| Status | In Review [ 10107 ] | In Progress [ 3 ] |
| Status | In Progress [ 3 ] | In Review [ 10107 ] |
Automated transition triggered when Adam Fraser merged pull request #3977 in GitHub -
| Status | In Review [ 10107 ] | Resolved [ 5 ] |
| Actual End | 2019-02-21 12:06 (issue has been closed) |
Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
CBG-220 Add size-based cap to rotated log files (#3977)
Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
CBG-220 Add size-based cap to rotated log files (#3977)
Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message:
Revert "CBG-220: Limit total size of rotated logs (#3972)" (#3976)
Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
CBG-220: Limit total size of rotated logs (#3972)
Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
CBG-220: Limit total size of rotated logs (#3972)
We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.