Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7413

Speedup incremental index updates, and make them generate less fragmentation

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.1
    • Component/s: view-engine
    • Security Level: Public
    • Labels:
      None

      Description

      Use very large on disk sorted batches, to reduce update time and fragmentation.
      Also benefit compaction retry phase.

      More important during rebalance or when user has automatic index updates disabled, and each indexer run processes very large number of changes.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        FilipeManana Filipe Manana (Inactive) added a comment - - edited

        http://review.couchbase.org/#/c/23674/

        From commit message, some benchmarks:

        "
        Test 1:

        Initially 3.2M documents, design document with 4 views (both
        from evperf lucky8 tests), index was built and later more
        3.2M documents were added (new 512 partitions added to the
        index as active).

        Initial build took: 11m17.911s
        Index file size after initial build: 380Mb

          • Before this change

        Incremental update took: 75m43.575s
        Index file size after incremental update: 30.0Gb

          • After this change

        Incremental update took: 18m36.762s
        Index file size after incremental update: 4.0Gb

        Test 2:

        Initially 3.2M documents (from lucky8 evperf test), design
        document with a single view (emits meta.id key and null value
        for each document), index was built and later more 3.2M documents
        were added (new 512 partitions added to the index as active).

        Initial build took: 6m23.199s
        Index file size after initial build: 134Mb

          • Before this change

        Incremental update took: 21m31.615s
        Index file size after incremental update: 8.6Gb

          • After this change

        Incremental update took: 7m13.394s
        Index file size after incremental update: 1.8Gb
        "

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - - edited http://review.couchbase.org/#/c/23674/ From commit message, some benchmarks: " Test 1: Initially 3.2M documents, design document with 4 views (both from evperf lucky8 tests), index was built and later more 3.2M documents were added (new 512 partitions added to the index as active). Initial build took: 11m17.911s Index file size after initial build: 380Mb Before this change Incremental update took: 75m43.575s Index file size after incremental update: 30.0Gb After this change Incremental update took: 18m36.762s Index file size after incremental update: 4.0Gb Test 2: Initially 3.2M documents (from lucky8 evperf test), design document with a single view (emits meta.id key and null value for each document), index was built and later more 3.2M documents were added (new 512 partitions added to the index as active). Initial build took: 6m23.199s Index file size after initial build: 134Mb Before this change Incremental update took: 21m31.615s Index file size after incremental update: 8.6Gb After this change Incremental update took: 7m13.394s Index file size after incremental update: 1.8Gb "
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        per bug scrub.

        are there more changes to be merged ? can you please resolve the issue once all has been reviewed and merged

        Show
        farshid Farshid Ghods (Inactive) added a comment - per bug scrub. are there more changes to be merged ? can you please resolve the issue once all has been reviewed and merged
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Maybe.
        I'll close it myself later.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Maybe. I'll close it myself later.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #548 (See http://qa.hq.northscale.net/job/github-couchdb-preview/548/)
        MB-7413 More efficient and faster incremental updates (Revision e64bba5b58f5a4f555b06e6a1cb9ee8cdb6992f7)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/include/couch_set_view.hrl
        • src/couch_set_view/test/08-deletes-cleanup.t
        • src/couch_set_view/test/19-compaction-retry.t
        • etc/couchdb/default.ini.tpl.in
        • src/couch_set_view/test/11-updates-cleanup-many-views.t
        • src/couch_set_view/test/10-updates-cleanup.t
        • src/couchdb/couch_btree.erl
        • src/couch_set_view/src/couch_set_view_group.erl
        • src/couch_set_view/src/couch_set_view_updater.erl
        • src/couch_set_view/src/couch_set_view_updater_helper.erl
        • src/couch_set_view/src/couch_set_view_compactor.erl
        • src/couch_set_view/src/couch_set_view_util.erl
        • src/couch_set_view/Makefile.am
        • src/couch_set_view/test/09-deletes-cleanup-many-views.t
        • src/couch_set_view/test/21-updater-cleanup.t
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #548 (See http://qa.hq.northscale.net/job/github-couchdb-preview/548/ ) MB-7413 More efficient and faster incremental updates (Revision e64bba5b58f5a4f555b06e6a1cb9ee8cdb6992f7) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/include/couch_set_view.hrl src/couch_set_view/test/08-deletes-cleanup.t src/couch_set_view/test/19-compaction-retry.t etc/couchdb/default.ini.tpl.in src/couch_set_view/test/11-updates-cleanup-many-views.t src/couch_set_view/test/10-updates-cleanup.t src/couchdb/couch_btree.erl src/couch_set_view/src/couch_set_view_group.erl src/couch_set_view/src/couch_set_view_updater.erl src/couch_set_view/src/couch_set_view_updater_helper.erl src/couch_set_view/src/couch_set_view_compactor.erl src/couch_set_view/src/couch_set_view_util.erl src/couch_set_view/Makefile.am src/couch_set_view/test/09-deletes-cleanup-many-views.t src/couch_set_view/test/21-updater-cleanup.t
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Merged.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Merged.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #552 (See http://qa.hq.northscale.net/job/github-couchdb-preview/552/)
        MB-7413 Remove duplicated checkpoint logic (Revision 8317c32c46455c17e7ce1570089848bd4663cce0)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/src/couch_set_view_updater.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #552 (See http://qa.hq.northscale.net/job/github-couchdb-preview/552/ ) MB-7413 Remove duplicated checkpoint logic (Revision 8317c32c46455c17e7ce1570089848bd4663cce0) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/src/couch_set_view_updater.erl
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #556 (See http://qa.hq.northscale.net/job/github-couchdb-preview/556/)
        MB-7413 Count # of seqs processed for initial build (Revision 7e8b81deddc4a3cc3747661c285d79ae025ba429)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/src/couch_set_view_updater.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #556 (See http://qa.hq.northscale.net/job/github-couchdb-preview/556/ ) MB-7413 Count # of seqs processed for initial build (Revision 7e8b81deddc4a3cc3747661c285d79ae025ba429) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/src/couch_set_view_updater.erl
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        evperf test results (from test added in CBD-767):

        jenkins: http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/62/console

        incremental update time: 5676.96956992 seconds

          • build 2.0.1-121 (when MB-7413 was first introduced)

        jenkins: http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/63/console

        incremental update time: 1812.51270294 seconds

          • build 2.0.1-140 (latest)

        jenkins: http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/64/console

        incremental update time: 1939.70245409

        Speedup confirmed in perf team's test environment.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - evperf test results (from test added in CBD-767): build 2.0.1-120 (before MB-7413 ) jenkins: http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/62/console incremental update time: 5676.96956992 seconds build 2.0.1-121 (when MB-7413 was first introduced) jenkins: http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/63/console incremental update time: 1812.51270294 seconds build 2.0.1-140 (latest) jenkins: http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/64/console incremental update time: 1939.70245409 Speedup confirmed in perf team's test environment.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #558 (See http://qa.hq.northscale.net/job/github-couchdb-preview/558/)
        MB-7413 Periodically call file:advise/4 (Revision 059b672d11d4ceaa39b8caa127e09a20558c5dc5)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/src/couch_set_view_updater_helper.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #558 (See http://qa.hq.northscale.net/job/github-couchdb-preview/558/ ) MB-7413 Periodically call file:advise/4 (Revision 059b672d11d4ceaa39b8caa127e09a20558c5dc5) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/src/couch_set_view_updater_helper.erl

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            FilipeManana Filipe Manana (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes