Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32068

Non linear increase in rebalance time when number of buckets increased

    XMLWordPrintable

Details

    Description

      Build 5.5.2-3733
      While doing multi bucket tests we observed that rebalance time is increasing non linearly wrt number of buckets.
      Test setup:
      Number of nodes: Initially 3, after rebalance 4
      Number of documents: 3 million (Total documents in all buckets)
      Key size: ~40b
      Average document size: 1024b
      Gets,Sets/sec: 500, 500(Addition of load on all buckets)
      Number of replicas: 1

      Test procedure:
      -Load documents in buckets
      -Start incremental load
      -After 15-20 mins rebalance in KV node
      -After rebalance continue load for 15-20 mins

      Here are observations:

      Logs are present at the end of console output on jenkins job.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          poonam Poonam Dhavale added a comment -   Links to the tests: 50 Bucket, 256 vBuckets:  http://perf.jenkins.couchbase.com/job/arke-multi-bucket/168/ 30 Bucket, 512 vBuckets:  http://perf.jenkins.couchbase.com/job/arke-multi-bucket/169/
          poonam Poonam Dhavale added a comment - - edited

           
          Also, ran couple of more tests:

          poonam Poonam Dhavale added a comment - - edited   Also, ran couple of more tests: 70 buckets, 128 vBuckets, rebalance took 4.6 minutes. (compared to 246 minutes with 1024 vBuckets) http://perf.jenkins.couchbase.com/job/arke-multi-bucket/171 /  30 buckets, 256 vBuckets, rebalance took 2.6 minutes http://perf.jenkins.couchbase.com/job/arke-multi-bucket/170 / 
          lynn.straus Lynn Straus added a comment -

          setting initial fix version to Mad Hatter so that investigation occurs in MH timeframe.  Please update fix version once investigation completes.

          lynn.straus Lynn Straus added a comment - setting initial fix version to Mad Hatter so that investigation occurs in MH timeframe.  Please update fix version once investigation completes.
          bo-chun.wang Bo-Chun Wang added a comment - - edited

          I have finished runs with different numbers of buckets. Each node has 24 CPU cores and 64GB memory. The memory quota is 45GB per node. 

           

          Build 7.0.0-4554

           

          Build 7.0.0-4797

           

          bo-chun.wang Bo-Chun Wang added a comment - - edited I have finished runs with different numbers of buckets. Each node has 24 CPU cores and 64GB memory. The memory quota is 45GB per node.    Build 7.0.0-4554 #Buckets Rebalance time(min) Jenkins job cbmonitor link CPU utilization (172.23.99.157) 10  1.3 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/6/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4554_rebalance_7842 during rebalance: 23% w/o rebalance: 8% 30  3.9 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/7/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4554_rebalance_a9d0 during rebalance: 28% w/o rebalance: 10% 50  7.8 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/8/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4554_rebalance_4559 during rebalance: 30% w/o rebalance: 13% 70  17.5 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/9/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4554_rebalance_e24c during rebalance: 30% w/o rebalance: 18% 100  50.0 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/10/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4554_rebalance_b15d during rebalance: 30% w/o rebalance: 20%   Build 7.0.0-4797 #Buckets Rebalance time(min) Jenkins job cbmonitor link CPU utilization (172.23.99.157) 10 1.3 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/11/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4797_rebalance_d30e during rebalance: 23% w/o rebalance: 8% 30 3.8 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/12/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4797_rebalance_0d87 during rebalance: 29% w/o rebalance: 10% 50 7.5 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/13/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4797_rebalance_5209 during rebalance: 32% w/o rebalance: 15% 70 16.0 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/14/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4797_rebalance_c912 during rebalance: 30% w/o rebalance: 18% 100  45.7 http://perf.jenkins.couchbase.com/job/themis-multibucket-kv-rebalance/15/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_700-4797_rebalance_c545 during rebalance: 30% w/o rebalance: 20%  

          I close this issue as the current performance results show significant improvement.

          bo-chun.wang Bo-Chun Wang added a comment - I close this issue as the current performance results show significant improvement.

          People

            bo-chun.wang Bo-Chun Wang
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty