Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.2.1
-
Untriaged
-
0
-
Unknown
Description
I am running a indexer rebalance test while runnig kv and scan workload in the background during the rebalance.
I ran it with 8000 ops/sec this time and 512 cbindexperf concurrency....unbounded scans
We ran it with 7.2.0-5330 (G.A) and 7.2.1-3878 ..
Scale out test 3-> 4 nodes
I can see that
The rebalance progress is stucked in 40% for 99%of the rebalance time and in just 5 seconds it jumped from 40% to 100%. The total rebalance time was 65.3 mins.We can clearly see this from timestamps below:
2023-07-30T20:23:01 [INFO] Rebalance progress: 40.00000000000001 %
|
2023-07-30T20:23:06 [INFO] Rebalance progress: 67.32673267326733 %
|
2023-07-30T20:23:07 [INFO] Rebalance progress: 98.34983498349834 %
|
2023-07-30T20:23:07 [INFO] Rebalance progress: 100 %
|
But 7.2.0 seems to increasing incrementally ..
Cbmointor comparison : http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=aether_721-5878_rebalance_3673&label=7.2.1&snapshot=aether_720-5330_rebalance_40d2&label=7.2.0
runs :
7.2.0-5330 : http://perf.jenkins.couchbase.com/job/aether/2733/consoleText
logs :
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.53.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.54.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.55.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.56.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.71.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.72.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/172.23.110.73.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/cpuprofile.prof
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2733/memprofile.prof
7.2.1-3878 : http://perf.jenkins.couchbase.com/job/aether/2734/consoleText
logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.53.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.54.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.55.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.56.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.71.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.72.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/172.23.110.73.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/cpuprofile.prof
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-aether-2734/memprofile.prof