Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
2.0.1, 2.1.0
-
Security Level: Public
-
None
-
Untriaged
Description
The size of the buffer used by the mergesort module is exclusively bounded by number of elements. This
is dangerous, because elements can have variable sizes, and
a small number of elements does not necessarily means that
the buffer size (byte size) is small.
Namely the treewriter module, used by the database compactor
to sort the temporary file containing records for the id btree,
was specifying a buffer element count of 100 * 1024 * 1024.
If for example there are 100 * 1024 * 1024 id records and each
has an average size of 512 bytes, the merge sort module buffers
50Gb of data in memory!
Although the id btree records are currently very small (under
a hundred bytes or so), use of other types of records may easily
cause too much memory consumption - this will be the case for
view records. Issue MB-8029 adds a module that uses the mergesort
module to sort files containing view records.