Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-220

Limit total size of rotated logs

    XMLWordPrintable

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Unresolved
    • 2.1.0
    • 2.6.0
    • SyncGateway
    • Security Level: Public
    • CBG Sprint 12, CBG Sprint 13
    • 8

    Description

      2.1 removed the option to clean up rotated log files based on number of backups (which could be directly related to file size with max_size*max_backups)

      Since we now gzip rotated log files, and have 4 separate log files, this calculation breaks down, and we can no longer rely on it to calculate disk usage.

      We need to have a way to limit the total size of logs, as a safeguard to prevent a node's disk from being completely filled up.

      As a bonus, we could look into removing the less important logs first (debug, info) to preserve errors and warnings.

      Migrated from: https://github.com/couchbase/sync_gateway/issues/3713

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          adamf Adam Fraser added a comment - - edited

          We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.

          adamf Adam Fraser added a comment - - edited We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.
          adamf Adam Fraser added a comment -

          Tasks:

          • Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level).  If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold.
          • Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation)
          • If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)
          adamf Adam Fraser added a comment - Tasks: Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level).  If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold. Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation) If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
          CBG-220 Add size-based cap to rotated log files (#3977)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message: CBG-220 Add size-based cap to rotated log files (#3977)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
          CBG-220 Add size-based cap to rotated log files (#3977)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message: CBG-220 Add size-based cap to rotated log files (#3977)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message:
          Revert "CBG-220: Limit total size of rotated logs (#3972)" (#3976)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message: Revert " CBG-220 : Limit total size of rotated logs (#3972)" (#3976)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
          CBG-220: Limit total size of rotated logs (#3972)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message: CBG-220 : Limit total size of rotated logs (#3972)

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
          CBG-220: Limit total size of rotated logs (#3972)

          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message: CBG-220 : Limit total size of rotated logs (#3972)

          People

            jacques.rascagneres Jacques Rascagneres
            ben.brooks Ben Brooks
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty