Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-220

Limit total size of rotated logs

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.6.0
    • Component/s: SyncGateway
    • Security Level: Public
    • Labels:
    • Sprint:
      CBG Sprint 12, CBG Sprint 13
    • Story Points:
      8

      Description

      2.1 removed the option to clean up rotated log files based on number of backups (which could be directly related to file size with max_size*max_backups)

      Since we now gzip rotated log files, and have 4 separate log files, this calculation breaks down, and we can no longer rely on it to calculate disk usage.

      We need to have a way to limit the total size of logs, as a safeguard to prevent a node's disk from being completely filled up.

      As a bonus, we could look into removing the less important logs first (debug, info) to preserve errors and warnings.

      Migrated from: https://github.com/couchbase/sync_gateway/issues/3713

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          adamf Adam Fraser added a comment - - edited

          We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.

          Show
          adamf Adam Fraser added a comment - - edited We delete rotated logs based on age today, so the main issue is handling scenarios where logs are being generated more quickly than they are aging out.
          Hide
          adamf Adam Fraser added a comment -

          Tasks:

          • Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level).  If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold.
          • Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation)
          • If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)
          Show
          adamf Adam Fraser added a comment - Tasks: Write a function that checks the total size of rotated logs on disk and compares to a threshold (per log level).  If threshold is exceeded, removes oldest (based on timestamp) until size is under 90% (80%?) of threshold. Investigate lumberjack to see if there's any way to trigger this function call on log rotation (or even identify the frequency of log rotation) If not, add a scheduled background job to run this on a relatively infrequent basis (hourly?)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
          CBG-220 Add size-based cap to rotated log files (#3977)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message: CBG-220 Add size-based cap to rotated log files (#3977)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message:
          CBG-220 Add size-based cap to rotated log files (#3977)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit a7edf70 with commit message: CBG-220 Add size-based cap to rotated log files (#3977)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message:
          Revert "CBG-220: Limit total size of rotated logs (#3972)" (#3976)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 1f884e3 with commit message: Revert " CBG-220 : Limit total size of rotated logs (#3972)" (#3976)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
          CBG-220: Limit total size of rotated logs (#3972)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message: CBG-220 : Limit total size of rotated logs (#3972)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message:
          CBG-220: Limit total size of rotated logs (#3972)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.6.0-2 contains sync_gateway commit 8d6f338 with commit message: CBG-220 : Limit total size of rotated logs (#3972)

            People

            Assignee:
            jacques.rascagneres Jacques Rascagneres
            Reporter:
            ben.brooks Ben Brooks
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty