Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-220

Limit total size of rotated logs

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Unresolved
    • 2.1.0
    • 2.6.0
    • SyncGateway
    • Security Level: Public
    • CBG Sprint 12, CBG Sprint 13
    • 8

    Description

      2.1 removed the option to clean up rotated log files based on number of backups (which could be directly related to file size with max_size*max_backups)

      Since we now gzip rotated log files, and have 4 separate log files, this calculation breaks down, and we can no longer rely on it to calculate disk usage.

      We need to have a way to limit the total size of logs, as a safeguard to prevent a node's disk from being completely filled up.

      As a bonus, we could look into removing the less important logs first (debug, info) to preserve errors and warnings.

      Migrated from: https://github.com/couchbase/sync_gateway/issues/3713

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          ben.brooks Ben Brooks created issue -
          ben.brooks Ben Brooks made changes -
          Field Original Value New Value
          Rank Ranked higher
          adamf Adam Fraser made changes -
          Assignee The One [ the one ] Ben Brooks [ ben.brooks ]
          adamf Adam Fraser made changes -
          Labels needs-info
          adamf Adam Fraser made changes -
          Story Points 8
          adamf Adam Fraser made changes -
          Sprint CBG Sprint 12 [ 749 ]
          adamf Adam Fraser made changes -
          Rank Ranked higher
          adamf Adam Fraser made changes -
          Planned End 2019-02-11 14:00 (started sprint)
          Planned Start 2019-02-11 14:00 (started sprint)
          adamf Adam Fraser added a comment - - edited

          We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.

          adamf Adam Fraser added a comment - - edited We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.
          adamf Adam Fraser added a comment -

          Tasks:

          • Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level).  If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold.
          • Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation)
          • If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)
          adamf Adam Fraser added a comment - Tasks: Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level).  If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold. Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation) If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)
          adamf Adam Fraser made changes -
          Assignee Ben Brooks [ ben.brooks ] Jacques Rascagneres [ jacques.rascagneres ]
          jacques.rascagneres Jacques Rascagneres made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          jacques.rascagneres Jacques Rascagneres made changes -
          Actual Start 2019-02-18 05:34 (issue has been started)
          jacques.rascagneres Jacques Rascagneres made changes -
          Status In Progress [ 3 ] In Review [ 10107 ]
          Automated transition triggered when Jacques Rascagneres created pull request #3972 in GitHub -
          Status In Review [ 10107 ] In Progress [ 3 ]
          jacques.rascagneres Jacques Rascagneres made changes -
          Status In Progress [ 3 ] In Review [ 10107 ]
          adamf Adam Fraser made changes -
          Sprint CBG Sprint 12 [ 749 ] CBG Sprint 12, CBG Sprint 13 [ 749, 753 ]
          adamf Adam Fraser made changes -
          Planned End 2019-02-11 14:00 2019-02-19 13:30 (started sprint)
          Planned Start 2019-02-11 14:00 2019-02-19 13:30 (started sprint)
          Automated transition triggered when Adam Fraser created pull request #3977 in GitHub -
          Status In Review [ 10107 ] In Progress [ 3 ]
          jacques.rascagneres Jacques Rascagneres made changes -
          Status In Progress [ 3 ] In Review [ 10107 ]
          Automated transition triggered when Adam Fraser merged pull request #3977 in GitHub -
          Status In Review [ 10107 ] Resolved [ 5 ]
          Anonymous made changes -
          Actual End 2019-02-21 12:06 (issue has been closed)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
          CBG-220 Add size-based cap to rotated log files (#3977)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message: CBG-220 Add size-based cap to rotated log files (#3977)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
          CBG-220 Add size-based cap to rotated log files (#3977)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message: CBG-220 Add size-based cap to rotated log files (#3977)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message:
          Revert "CBG-220: Limit total size of rotated logs (#3972)" (#3976)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message: Revert " CBG-220 : Limit total size of rotated logs (#3972)" (#3976)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
          CBG-220: Limit total size of rotated logs (#3972)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message: CBG-220 : Limit total size of rotated logs (#3972)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
          CBG-220: Limit total size of rotated logs (#3972)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message: CBG-220 : Limit total size of rotated logs (#3972)

          People

            jacques.rascagneres Jacques Rascagneres
            ben.brooks Ben Brooks
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty