Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8521

Improve rebalance speed of small dataset

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.1
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None

      Description

      There are a few but very common rebalance use cases that our customers and prospective customers are performing which show very slow speed and then lead to a poor perception of the software's performance. At larger scale, the rebalance speed can be easier explain by many items and lots of data

      These use cases include:
      -4 nodes to 6 nodes with 8k ops/sec on one of the beer-sample database with the gamesim sample database installed but not under load. This is the demo that our sales team gives on a very regular basis.
      -1 node to 2 nodes with an empty bucket. This is by far the most common, and our users expect that it should be very fast since there is no data...it's not very fast.
      -1 node to 2 nodes with a small bucket. Same as above, only slightly less common

      Ideally any of the above use cases should show the rebalance progress moving very quickly, and take no more than a few minutes (less than 10).

      Let me know if there is more specific information required, or a need for an environment to be provided

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          perry Perry Krug added a comment -

          The empty bucket use case is definitely a common and painful one, but I would not want us to implement something specfically and only applicable to a bucket with no items if possible. Even with very small datasets (a few thousand items, like our sample datasets) the rebalance should be very very fast in the minds of our customers.

          Thanks!

          Show
          perry Perry Krug added a comment - The empty bucket use case is definitely a common and painful one, but I would not want us to implement something specfically and only applicable to a bucket with no items if possible. Even with very small datasets (a few thousand items, like our sample datasets) the rebalance should be very very fast in the minds of our customers. Thanks!
          Hide
          chiyoung Chiyoung Seo added a comment -

          With the above fix, there is no more regressions in rebalance speed for the empty bucket or small dataset bucket.

          I compared the rebalance speed (1->2 reblanace) between 2.0.1 and 2.2.0 with the above fix:

          2.0.1

          empty bucket - 3 min 9 sec

          bucket with 100K items - 3 min 40 sec

          2.2.0 with the above fix

          empty bucket - 3 min

          bucket with 100K items - 3 min 17 sec

          One of the main reasons is the well-known fsync issue because we have a separate database file per partition.

          However, we all agree that we should make the rebalance speed of empty or small dataset bucket much faster. I need to figure out the further optimizations in the rebalance path.

          Show
          chiyoung Chiyoung Seo added a comment - With the above fix, there is no more regressions in rebalance speed for the empty bucket or small dataset bucket. I compared the rebalance speed (1->2 reblanace) between 2.0.1 and 2.2.0 with the above fix: 2.0.1 empty bucket - 3 min 9 sec bucket with 100K items - 3 min 40 sec 2.2.0 with the above fix empty bucket - 3 min bucket with 100K items - 3 min 17 sec One of the main reasons is the well-known fsync issue because we have a separate database file per partition. However, we all agree that we should make the rebalance speed of empty or small dataset bucket much faster. I need to figure out the further optimizations in the rebalance path.
          Hide
          chiyoung Chiyoung Seo added a comment -

          Per discussion with Dipti / Maria, we will create a new ticket for the rebalance time improvement that is targeted for 2.2.0 release.

          Show
          chiyoung Chiyoung Seo added a comment - Per discussion with Dipti / Maria, we will create a new ticket for the rebalance time improvement that is targeted for 2.2.0 release.
          Hide
          thuan Thuan Nguyen added a comment -

          Tested on build 2.1.1-763
          ubuntu 12.04 64bit 4GB RAM, 4 core cpu

          • Empty default bucket
          • Rebalance 1 ==> 2 : 2 min 15 sec
          • Rebalance 2 ==> 3 : 2 min 21 sec
          • Rebalance 3 ==> 4 : 1 min 12 sec
          • 100K keys with size 128 bytes in default bucket
          • Rebalance 1 ==> 2 : 2 min 55 sec
          • Rebalance 2 ==> 3 : 2 min 58 sec
          • Rebalance 3 ==> 4 : 1 min 9 sec

          centos 5.8 64-bit 4GB RAM, 4 core cpu

          • Empty default bucket
          • Rebalance 1 ==> 2 : 2 min 25 sec
          • Rebalance 2 ==> 3 : 2 min 26 sec
          • Rebalance 3 ==> 4 : 1 min 6 sec
          • 100K keys with size 128 bytes in default bucket
          • Rebalance 1 ==> 2 : 2 min 58 sec
          • Rebalance 2 ==> 3 : 2 min 52 sec
          • Rebalance 3 ==> 4 : 1 min 16 sec

          windows 2008 R2 64-bit 4GB RAM, 4 core cpu

          • Empty default bucket
          • Rebalance 1 ==> 2 : 3 min 40 sec
          • Rebalance 2 ==> 3 : 7 min 30 sec
          • Rebalance 3 ==> 4 : 4 min 21 sec
          • 100K keys with size 128 bytes in default bucket
          • Rebalance 1 ==> 2 : 4 min 28 sec
          • Rebalance 2 ==> 3 : 5 min 52 sec
          • Rebalance 3 ==> 4 : 5 min 10 sec
          Show
          thuan Thuan Nguyen added a comment - Tested on build 2.1.1-763 ubuntu 12.04 64bit 4GB RAM, 4 core cpu Empty default bucket Rebalance 1 ==> 2 : 2 min 15 sec Rebalance 2 ==> 3 : 2 min 21 sec Rebalance 3 ==> 4 : 1 min 12 sec 100K keys with size 128 bytes in default bucket Rebalance 1 ==> 2 : 2 min 55 sec Rebalance 2 ==> 3 : 2 min 58 sec Rebalance 3 ==> 4 : 1 min 9 sec centos 5.8 64-bit 4GB RAM, 4 core cpu Empty default bucket Rebalance 1 ==> 2 : 2 min 25 sec Rebalance 2 ==> 3 : 2 min 26 sec Rebalance 3 ==> 4 : 1 min 6 sec 100K keys with size 128 bytes in default bucket Rebalance 1 ==> 2 : 2 min 58 sec Rebalance 2 ==> 3 : 2 min 52 sec Rebalance 3 ==> 4 : 1 min 16 sec windows 2008 R2 64-bit 4GB RAM, 4 core cpu Empty default bucket Rebalance 1 ==> 2 : 3 min 40 sec Rebalance 2 ==> 3 : 7 min 30 sec Rebalance 3 ==> 4 : 4 min 21 sec 100K keys with size 128 bytes in default bucket Rebalance 1 ==> 2 : 4 min 28 sec Rebalance 2 ==> 3 : 5 min 52 sec Rebalance 3 ==> 4 : 5 min 10 sec
          Hide
          kzeller kzeller added a comment -

          Hi Tony,

          Added to RN 2.1.1. Let me know if that is all we need, or if we need more content updates in docs.

          Thanks,

          Karen

          Show
          kzeller kzeller added a comment - Hi Tony, Added to RN 2.1.1. Let me know if that is all we need, or if we need more content updates in docs. Thanks, Karen

            People

            • Assignee:
              thuan Thuan Nguyen
              Reporter:
              perry Perry Krug
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes