Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8054

Couchstore's mergesort module, currently used for db compaction, can buffer too much data in memory

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • bug-backlog
    • 2.0.1, 2.1.0
    • storage-engine
    • Security Level: Public
    • None
    • Untriaged

    Description

      The size of the buffer used by the mergesort module is exclusively bounded by number of elements. This
      is dangerous, because elements can have variable sizes, and
      a small number of elements does not necessarily means that
      the buffer size (byte size) is small.

      Namely the treewriter module, used by the database compactor
      to sort the temporary file containing records for the id btree,
      was specifying a buffer element count of 100 * 1024 * 1024.
      If for example there are 100 * 1024 * 1024 id records and each
      has an average size of 512 bytes, the merge sort module buffers
      50Gb of data in memory!

      Although the id btree records are currently very small (under
      a hundred bytes or so), use of other types of records may easily
      cause too much memory consumption - this will be the case for
      view records. Issue MB-8029 adds a module that uses the mergesort
      module to sort files containing view records.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            srinath.duvuru Srinath Duvuru
            FilipeManana Filipe Manana (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty