Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25631

Improve statistics / logging from compaction

    XMLWordPrintable

Details

    Description

      During investigation of slow background fetches in a customer environment, correlation was found with compaction runs. However currently there's limited insight into what compaction is doing from our log files - for example all we see the start time and end time:

      2017-06-28T05:22:23.134793Z NOTICE (bucket) Compaction of db file id: 228 scheduled (awaiting completion).
      2017-06-28T05:22:37.969960Z NOTICE (bucket) Compaction of db file id: 228 completed.
      

      It would assist in assessing the impact of compaction if additional details could be logged - for example:

      • Size of the input file
      • Fragmentation ratio of the input file / size of useful data in the input file.
      • Size of the output file
      • Number of documents copied into the new file
      • Number of documents discarded due to expiration during compaction
      • Number of tombstones purged during compaction
      • Sequence number which we purged up to.

      If the information is already recorded by couchstore then we just need to add it to our logging in kv_engine / couch_compact; if not then further work may be required in couchstore to expose them.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jwalker Jim Walker added a comment - - edited

          Back to vulcan as this could really help current cases, see linked CBSE where some of this logging could expedite the RCA.

          Note we'll see how many of these stats can be implemented in a reasonable timeframe

          jwalker Jim Walker added a comment - - edited Back to vulcan as this could really help current cases, see linked CBSE where some of this logging could expedite the RCA. Note we'll see how many of these stats can be implemented in a reasonable timeframe
          jwalker Jim Walker added a comment -

          Expired items is trickier as the decision to expire an item occurs many hops away from compaction, we do though have ep_expired_compactor for some visibility into expiry via compaction

          jwalker Jim Walker added a comment - Expired items is trickier as the decision to expire an item occurs many hops away from compaction, we do though have ep_expired_compactor for some visibility into expiry via compaction

          Build 5.5.0-1673 contains kv_engine commit fdd7414e1fd2dfdce112317f39cbdf4df9347369 with commit message:
          MB-25631: Log more details about compaction
          https://github.com/couchbase/kv_engine/commit/fdd7414e1fd2dfdce112317f39cbdf4df9347369

          build-team Couchbase Build Team added a comment - Build 5.5.0-1673 contains kv_engine commit fdd7414e1fd2dfdce112317f39cbdf4df9347369 with commit message: MB-25631 : Log more details about compaction https://github.com/couchbase/kv_engine/commit/fdd7414e1fd2dfdce112317f39cbdf4df9347369

          Build couchbase-server-5.5.0-1673 contains kv_engine commit fdd7414e1fd2dfdce112317f39cbdf4df9347369 with commit message:
          MB-25631: Log more details about compaction
          https://github.com/couchbase/kv_engine/commit/fdd7414e1fd2dfdce112317f39cbdf4df9347369

          build-team Couchbase Build Team added a comment - Build couchbase-server-5.5.0-1673 contains kv_engine commit fdd7414e1fd2dfdce112317f39cbdf4df9347369 with commit message: MB-25631 : Log more details about compaction https://github.com/couchbase/kv_engine/commit/fdd7414e1fd2dfdce112317f39cbdf4df9347369
          drigby Dave Rigby added a comment -

          Example of improved log output:

          2018-02-02T15:03:23.450336Z INFO (default) Compaction of db file id: 29 completed (ok). tombstones_purged:0, collection_items_erased:0, pre{size:258139, items:103, deleted_items:0, purge_seqno:0}, post{size:24667, items:103, deleted_items:0, purge_seqno:0}
          

          drigby Dave Rigby added a comment - Example of improved log output: 2018-02-02T15:03:23.450336Z INFO (default) Compaction of db file id: 29 completed (ok). tombstones_purged:0, collection_items_erased:0, pre{size:258139, items:103, deleted_items:0, purge_seqno:0}, post{size:24667, items:103, deleted_items:0, purge_seqno:0}

          People

            drigby Dave Rigby
            drigby Dave Rigby
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty