Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7188

Improve incremental indexing and cleanup performance, make fragmentation grow significantly more slowly, and overall better performance

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Fixed
    • Major
    • 2.0
    • None
    • view-engine
    • Security Level: Public
    • None

    Description

      By prefixing keys for the id btree with the vbucket Id, incremental index updates, especially large ones such as those after/during a rebalance, get significantly faster with the added benefit of generating less file fragmentation. Also it speeds up cleanup and better overall performance (due to slower fragmentation, less compactions).

      Example test, using lucky6.conf evperf dataset. Index was initially built with 0.5M documents and then 512 new vbuckets were added, which brought 1M new documents:

      Before

      initial build: 3m55.637s (final file size 85Mb)
      incremental update: 54m47.857s (final file size 11Gb)

      with patch

      initial build: 3m59.553s (final file size 89 Mb)
      incremental update: 45m1.006s (final file size 8.7Gb)

      Gains even higher, when used together with MB-7131, example for a small test of 0.5M + 0.5M documents:

      Before

      initial build: 3m58.688s ( final file size 85 Mb )
      incremental update: 20m11.722s ( final file size 4967 Mb )

      After

      initial build: 3m57.042s ( final file size 89Mb )
      incremental update: 16m0.273s ( final file size 4109Mb )

      After + with MB-7131

      initial build: 3m59.265s ( final file size 89 Mb )
      incremental update: 15m10.753s ( final file size 3819 Mb )

      All the tests naturally had view compaction not enabled.
      Gains more significant for larger datasets.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            FilipeManana Filipe Manana (Inactive)
            FilipeManana Filipe Manana (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty