Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7522

Optimize very common index state transitions (rebalance)

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.1
    • Component/s: view-engine
    • Security Level: Public
    • Labels:
      None
    • Flagged:
      Release Note

      Description

      Optimize 2 types of state transitions:

      1) If a state transition consists only of moving partitions
      between the passive and active states (or vice-versa), that
      is, no new partitions are added nor existing partitions are
      marked for cleanup, don't restart the updater (at the expense
      of more complex logic to correct snapshots received from a
      currently running updater).

      2) If a state transition only adds new partitions to the
      passive state (as ns_server's rebalance frequently calls),
      don't restart the updater, and send the updater the new
      list of passive partitions, so that it has a chance to
      iterate over those partitions' content and index them in
      the same run.

      As long as the rebalancer in ns_server doesn't keep changing
      its logic regarding state transitions all the time, this
      provides a significant reduction of the indexing time. However
      for at least some scenarios, there's still significant portion
      of rebalance time not spent on indexing nor index compaction.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #552 (See http://qa.hq.northscale.net/job/github-couchdb-preview/552/)
        MB-7522 Do updater restarts if a pending transition exists (Revision 060f7ae833dd871af56d188490a6371f35509782)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/src/couch_set_view_group.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #552 (See http://qa.hq.northscale.net/job/github-couchdb-preview/552/ ) MB-7522 Do updater restarts if a pending transition exists (Revision 060f7ae833dd871af56d188490a6371f35509782) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/src/couch_set_view_group.erl
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #551 (See http://qa.hq.northscale.net/job/github-couchdb-preview/551/)
        MB-7522 Further test improvements/coverage (Revision 8deb85624bacdfa290adb4973f7ba1d204885309)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #551 (See http://qa.hq.northscale.net/job/github-couchdb-preview/551/ ) MB-7522 Further test improvements/coverage (Revision 8deb85624bacdfa290adb4973f7ba1d204885309) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #550 (See http://qa.hq.northscale.net/job/github-couchdb-preview/550/)
        MB-7522 Fix race condition in test (Revision 84d25c7cb136c9f66adbb572c99cca81235ef13e)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #550 (See http://qa.hq.northscale.net/job/github-couchdb-preview/550/ ) MB-7522 Fix race condition in test (Revision 84d25c7cb136c9f66adbb572c99cca81235ef13e) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #549 (See http://qa.hq.northscale.net/job/github-couchdb-preview/549/)
        MB-7522 Improve test 24-updater-add-more-passive-partitions.t (Revision 9126caff6319243a9b0fdfa50e05477c94092d00)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #549 (See http://qa.hq.northscale.net/job/github-couchdb-preview/549/ ) MB-7522 Improve test 24-updater-add-more-passive-partitions.t (Revision 9126caff6319243a9b0fdfa50e05477c94092d00) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Hide
        kzeller kzeller added a comment -

        I'm elevating this to 2.0.1 release notes....

        Show
        kzeller kzeller added a comment - I'm elevating this to 2.0.1 release notes....
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Merged.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Merged.
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Gerrit: http://review.couchbase.org/#/c/23871/

        Some results: https://docs.google.com/document/d/1aHHhY-ami84-aEQyx42h95rlp7Vi6WTwlozaIstFgfA/edit , related to MB-6726.

        Drops all needs to restart updater for the current type of transitions the rebalancer does to indexes.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Gerrit: http://review.couchbase.org/#/c/23871/ Some results: https://docs.google.com/document/d/1aHHhY-ami84-aEQyx42h95rlp7Vi6WTwlozaIstFgfA/edit , related to MB-6726 . Drops all needs to restart updater for the current type of transitions the rebalancer does to indexes.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #548 (See http://qa.hq.northscale.net/job/github-couchdb-preview/548/)
        MB-7522 Optimize index state transitions (rebalance) (Revision ac82c60302422747e8804d566211a20684ec78fb)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/test/17-unindexable-partitions.t
        • src/couch_set_view/Makefile.am
        • src/couch_set_view/src/couch_set_view_group.erl
        • src/couch_set_view/test/21-updater-cleanup.t
        • src/couch_set_view/src/couch_set_view_updater.erl
        • src/couch_set_view/src/couch_set_view_util.erl
        • src/couch_set_view/test/19-compaction-retry.t
        • src/couch_set_view/test/05-replicas-transfer.t
        • src/couch_set_view/test/07-replica-compaction.t
        • src/couch_set_view/test/24-updater-add-more-passive-partitions.t
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #548 (See http://qa.hq.northscale.net/job/github-couchdb-preview/548/ ) MB-7522 Optimize index state transitions (rebalance) (Revision ac82c60302422747e8804d566211a20684ec78fb) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/test/17-unindexable-partitions.t src/couch_set_view/Makefile.am src/couch_set_view/src/couch_set_view_group.erl src/couch_set_view/test/21-updater-cleanup.t src/couch_set_view/src/couch_set_view_updater.erl src/couch_set_view/src/couch_set_view_util.erl src/couch_set_view/test/19-compaction-retry.t src/couch_set_view/test/05-replicas-transfer.t src/couch_set_view/test/07-replica-compaction.t src/couch_set_view/test/24-updater-add-more-passive-partitions.t

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            FilipeManana Filipe Manana (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes