Details
-
Improvement
-
Resolution: Fixed
-
Major
-
2.0, 2.0.1, 2.1.0
-
Security Level: Public
-
None
Description
Move the KV batch sorting and file merging/sorting into a C process (using couchstore). Erlang will just populate files with unsorted KV files and then use this external C process to do the sorting and merging, as well as later building a btree bottom up from a sorted file (initial index build).
This will make index builds and updates more efficient, and open as well other optimization possibilities for later.
Sorting batches of KVs (in memory) and merging KV files (on disk) currently takes more than 60% of the time for initial index builds and at least 50% of the time for incremental index updates.
In a later phase, index btree updates and cleanup will also be performed externally via the C process, further improving efficiency and speeding up index updates, and keeping a minimal work in Erlang (mostly management tasks).
Moving this into an external process will also likely result in less resource consumption (CPU, memory, as Erlang consumes too much of them) and have a positive impact on everything else running inside the Erlang VM (queries, etc).
Attachments
Gerrit Reviews
For Gerrit Dashboard: MB-8029 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
25588,2 | MB-8054 Bound buffer size for merge sorts | master | couchstore | Status: ABANDONED | +2 | +1 |
25590,4 | MB-8029 Make json collator input parameters const | master | couchstore | Status: MERGED | +2 | +1 |
25591,9 | MB-8029 Add view file sorter for incremental updates | master | couchstore | Status: MERGED | +2 | +1 |
25592,3 | MB-8029 Use native file sorter for incremental index updates | master | couchdb | Status: MERGED | +2 | +1 |
25635,2 | MB-8029 Remove old (partial) indexer code | master | couchstore | Status: MERGED | +2 | +1 |
25793,3 | MB-8029 Add file_merger module | master | couchstore | Status: MERGED | +2 | +1 |
25794,3 | MB-8029 Add view file merger module | master | couchstore | Status: MERGED | +2 | +1 |
25795,4 | MB-8029 Add program to merge view files | master | couchstore | Status: MERGED | +2 | +1 |
25798,2 | MB-8029 Use native file merger for index builds | master | couchdb | Status: MERGED | +2 | +1 |
26189,3 | MB-8029 Add file sorter module | master | couchstore | Status: MERGED | +2 | +1 |
26190,2 | MB-8029 Make view file sorter use new file sorter module | master | couchstore | Status: MERGED | +2 | +1 |
26191,2 | MB-8029 Refactor view file merging callbacks | master | couchstore | Status: MERGED | +2 | +1 |
26192,2 | MB-8029 View file sorter can now sort initial build files | master | couchstore | Status: MERGED | +2 | +1 |
26193,2 | MB-8029 Pass tmp dir to view file sorter command | master | couchdb | Status: MERGED | +2 | +1 |
26194,2 | MB-8029 More efficient and simpler index build | master | couchdb | Status: MERGED | +2 | +1 |
29031,2 | MB-8029 Avoid unnecessary endianness convertions | master | couchstore | Status: MERGED | +2 | +1 |
29034,3 | MB-8029 Avoid unnecessary endianness convertions | master | couchdb | Status: MERGED | +2 | +1 |