Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34286

Indexer log should not be dominated by huge stats blocks

    XMLWordPrintable

Details

    Description

      At present, the indexer logs stats as a JSON object within a normal log line. This becomes impenetrable, as the number of indexes on this cluster grows. I realise different people have different workflows and use different tools, but taking less as an example of a reasonably robust way of viewing logs essentially makes scrolling through the logs impossible without first removing these lines.


      Taking the sizings, we can see that 325MB of this file (~85%) is these stats blocks:

      $ grep 'PeriodicStats' ns_server.indexer.log > indexer_stats.log; ls -lh indexer_stats.log ns_server.indexer.log
      -rw-r--r--  1 jflath  staff   325M 22 May 16:08 indexer_stats.log
      -rw-rw----  1 jflath  staff   380M 22 May 12:49 ns_server.indexer.log
      

      Don't get me wrong, I love stats, but this is not a suitable or supportable way to log them as the index count increases. The logging guidelines strongly recommend that stats aren't dumped into the "actual" logs for this reason.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          The other (more problematic side) of this is that Indexer logs only usually last for a few hours at best (depending on number of indexes) because these huge stat blocks cause the logs to rotate exceptionally quickly.

          matt.carabine Matt Carabine added a comment - The other (more problematic side) of this is that Indexer logs only usually last for a few hours at best (depending on number of indexes) because these huge stat blocks cause the logs to rotate exceptionally quickly.

          Following improvements are done for this.

          1. Separate log file for indexer stats.
          2. Stats prefix deduplication.
          3. Stats value deduplication.

          MB-31935.

          Closing this MB.

          amit.kulkarni Amit Kulkarni added a comment - Following improvements are done for this. 1. Separate log file for indexer stats. 2. Stats prefix deduplication. 3. Stats value deduplication. MB-31935 . Closing this MB.

          Verified in the logs generated by system and volume tests where are there are large number of indexes. Stats are logged to indexer_stats.log file and not indexer.log.

          mihir.kamdar Mihir Kamdar (Inactive) added a comment - Verified in the logs generated by system and volume tests where are there are large number of indexes. Stats are logged to indexer_stats.log file and not indexer.log.

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            James Flather James Flather
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty