Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-752

sgcollect: Re-gzip rotated logfiles post-redaction, clean up intermediate extracted files

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 2.8.0
    • None
    • SyncGateway
    • Security Level: Public
    • None
    • CBG Sprint 44
    • 3

    Description

      When we extract rotated log files, we extract them to perform log redaction on them, and leave them in an extracted state lying around in /tmp for the duration of the sgcollect_info run before putting them into the final zip.

      This seems to be counter to what was done in CBG-244, but here's an example with around 6GB of pre-compression logs:

       

      There are a couple of things we can do to optimise the collection of rotated logs:

      1. If we're not performing log redaction, skip the extraction step completely. Put the .gzip files straight into the zip file as-is.
      2. If we are performing redaction:
        1. extract
        2. redact
        3. re-compress
        4. put the newly redacted/compressed file into the zip file
        5. Make sure the extracted files are cleaned up as we iterate over each log file, to reduce the overall disk requirements.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jacques.rascagneres Jacques Rascagneres
            ben.brooks Ben Brooks
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty