Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8029

Optimize index builds and updates by moving some work to C (couchstore)

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 3.0
    • 2.0, 2.0.1, 2.1.0
    • view-engine
    • Security Level: Public
    • None

    Description

      Move the KV batch sorting and file merging/sorting into a C process (using couchstore). Erlang will just populate files with unsorted KV files and then use this external C process to do the sorting and merging, as well as later building a btree bottom up from a sorted file (initial index build).

      This will make index builds and updates more efficient, and open as well other optimization possibilities for later.

      Sorting batches of KVs (in memory) and merging KV files (on disk) currently takes more than 60% of the time for initial index builds and at least 50% of the time for incremental index updates.
      In a later phase, index btree updates and cleanup will also be performed externally via the C process, further improving efficiency and speeding up index updates, and keeping a minimal work in Erlang (mostly management tasks).

      Moving this into an external process will also likely result in less resource consumption (CPU, memory, as Erlang consumes too much of them) and have a positive impact on everything else running inside the Erlang VM (queries, etc).

      Attachments

        For Gerrit Dashboard: MB-8029
        # Subject Branch Project Status CR V

        Activity

          People

            ketaki Ketaki Gangal (Inactive)
            FilipeManana Filipe Manana (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty