Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58102

Rebalance progress is not showing incremental growth

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      I am running a indexer rebalance test while runnig kv and scan workload in the background during the rebalance.
      I ran it with 8000 ops/sec this time and 512 cbindexperf concurrency....unbounded scans 
      We ran it with 7.2.0-5330 (G.A) and 7.2.1-3878 .. 
      Scale out test 3-> 4 nodes 
      I can see that 

      The rebalance progress is stucked in 40% for 99%of the rebalance time and in just 5 seconds it jumped from 40% to 100%. The total rebalance time was 65.3 mins.We can clearly see this from timestamps below:

      2023-07-30T20:23:01 [INFO] Rebalance progress: 40.00000000000001 %

      2023-07-30T20:23:06 [INFO] Rebalance progress: 67.32673267326733 %

      2023-07-30T20:23:07 [INFO] Rebalance progress: 98.34983498349834 %

      2023-07-30T20:23:07 [INFO] Rebalance progress: 100 % 

      But 7.2.0 seems to increasing incrementally ..

      Cbmointor comparison : http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=aether_721-5878_rebalance_3673&label=7.2.1&snapshot=aether_720-5330_rebalance_40d2&label=7.2.0
      runs : 
      7.2.0-5330 : http://perf.jenkins.couchbase.com/job/aether/2733/consoleText 
      logs :
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.53.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.54.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.55.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.56.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.71.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.72.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.73.zip  
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/cpuprofile.prof 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/memprofile.prof 
      7.2.1-3878 : http://perf.jenkins.couchbase.com/job/aether/2734/consoleText 

      logs: 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.53.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.54.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.55.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.56.zip  
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.71.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.72.zip  
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.73.zip 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/cpuprofile.prof 
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/memprofile.prof 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              devansh.srivastava Devansh Srivastava
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty