Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7188

Improve incremental indexing and cleanup performance, make fragmentation grow significantly more slowly, and overall better performance

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0
    • Component/s: view-engine
    • Security Level: Public
    • Labels:
      None

      Description

      By prefixing keys for the id btree with the vbucket Id, incremental index updates, especially large ones such as those after/during a rebalance, get significantly faster with the added benefit of generating less file fragmentation. Also it speeds up cleanup and better overall performance (due to slower fragmentation, less compactions).

      Example test, using lucky6.conf evperf dataset. Index was initially built with 0.5M documents and then 512 new vbuckets were added, which brought 1M new documents:

      Before

      initial build: 3m55.637s (final file size 85Mb)
      incremental update: 54m47.857s (final file size 11Gb)

      with patch

      initial build: 3m59.553s (final file size 89 Mb)
      incremental update: 45m1.006s (final file size 8.7Gb)

      Gains even higher, when used together with MB-7131, example for a small test of 0.5M + 0.5M documents:

      Before

      initial build: 3m58.688s ( final file size 85 Mb )
      incremental update: 20m11.722s ( final file size 4967 Mb )

      After

      initial build: 3m57.042s ( final file size 89Mb )
      incremental update: 16m0.273s ( final file size 4109Mb )

      After + with MB-7131

      initial build: 3m59.265s ( final file size 89 Mb )
      incremental update: 15m10.753s ( final file size 3819 Mb )

      All the tests naturally had view compaction not enabled.
      Gains more significant for larger datasets.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        it will be very interesting to run the 40M items test with these fixes and compare index file size and indexing speed as reported by active_tasks

        Show
        farshid Farshid Ghods (Inactive) added a comment - it will be very interesting to run the 40M items test with these fixes and compare index file size and indexing speed as reported by active_tasks
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Wouldn't call it a fix, but rather an improvement, as it doesn't address any functionality issue

        Yes, larger datasets and design documents with several views is where more difference is noticeable, specially together with MB-7131.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Wouldn't call it a fix, but rather an improvement, as it doesn't address any functionality issue Yes, larger datasets and design documents with several views is where more difference is noticeable, specially together with MB-7131 .
        Hide
        steve Steve Yen added a comment -

        fixes are now on gerrit

        Show
        steve Steve Yen added a comment - fixes are now on gerrit
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #537 (See http://qa.hq.northscale.net/job/github-couchdb-preview/537/)
        MB-7188 Prefix id btree keys with partition id (Revision 8befa9b174ae250c51a22c84de9cfd55b96bb4bc)

        Result = SUCCESS
        Farshid Ghods :
        Files :

        • src/couch_set_view/test/13-progressive-cleanup.t
        • src/couch_set_view/test/17-unindexable-partitions.t
        • src/couch_set_view/test/09-deletes-cleanup-many-views.t
        • src/couch_set_view/test/14-duplicated-keys-per-doc.t
        • src/couch_set_view/test/05-replicas-transfer.t
        • src/couch_set_view/src/couch_set_view_updater.erl
        • src/couch_set_view/test/08-deletes-cleanup.t
        • src/couch_set_view/test/19-compaction-retry.t
        • src/couch_set_view/test/15-passive-partitions.t
        • src/couch_set_view/test/16-pending-transition.t
        • src/couch_set_view/test/22-compactor-cleanup.t
        • src/couch_set_view/test/21-updater-cleanup.t
        • src/couch_set_view/test/11-updates-cleanup-many-views.t
        • src/couch_set_view/test/10-updates-cleanup.t
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #537 (See http://qa.hq.northscale.net/job/github-couchdb-preview/537/ ) MB-7188 Prefix id btree keys with partition id (Revision 8befa9b174ae250c51a22c84de9cfd55b96bb4bc) Result = SUCCESS Farshid Ghods : Files : src/couch_set_view/test/13-progressive-cleanup.t src/couch_set_view/test/17-unindexable-partitions.t src/couch_set_view/test/09-deletes-cleanup-many-views.t src/couch_set_view/test/14-duplicated-keys-per-doc.t src/couch_set_view/test/05-replicas-transfer.t src/couch_set_view/src/couch_set_view_updater.erl src/couch_set_view/test/08-deletes-cleanup.t src/couch_set_view/test/19-compaction-retry.t src/couch_set_view/test/15-passive-partitions.t src/couch_set_view/test/16-pending-transition.t src/couch_set_view/test/22-compactor-cleanup.t src/couch_set_view/test/21-updater-cleanup.t src/couch_set_view/test/11-updates-cleanup-many-views.t src/couch_set_view/test/10-updates-cleanup.t
        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Merged: http://review.couchbase.org/#/c/22559/

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            FilipeManana Filipe Manana (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes