Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38027

Investigate why Rift fails to outperform SQLite with large documents

    XMLWordPrintable

Details

    Description

      Motivation:
      After Carlos Gonzalez Betancort fixed the storage benchmarks we noticed that while Rift does outperform SQLite with smaller documents (which is what we primarily test and benchmark with) it struggles when we start backing up larger documents.

      Possible causes:
      1) Unnecessary stress on the garbage collector; our checkSumAndWrite function allocates more memory that is specifically required (this was to avoid dynamic allocation when copying over slice values which is far more expensive). I've written some benchmarks testing various different solutions and believe that I have come up with a solution which is both quick, and avoids allocating additional memory (I achieved this by using the crc32 libraries 'Update' function which allows us to checksum the various parts of the document data without needing to do it in one go).
      2) Although not necessarily a direct cause to this, I noticed that the Rift meta/data ingestion streams are almost always empty i.e. data is being processed very quickly after handed to the Rift workers (the channel is set to a buffer value of five). This indicates that we should investigate whether it's worth having the additional (resource) overhead and code complexity of having the meta/data writers be asynchronous.

      3) Rift periodically syncs data to disk (using 'file.Sync') this is extremely expensive and won't be unnecessary if we avoid using an asynchronous worker model (the data stored should be equal or ahead of the data in the snapshot files). Removing the periodic sync (but keeping the periodic buffer flush/transaction commit) means that we maintain the same guarantees that we make with SQLite; your new backup is safe barring a power loss or os crash.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            james.lee James Lee
            james.lee James Lee
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty