Details
-
Bug
-
Resolution: Fixed
-
Major
-
3.0
-
Security Level: Public
-
Build 3.0.0-773
Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630 (24 vCPU)
Memory = 64 GB
Disk = RAID 10 HDD
-
Untriaged
-
Centos 64-bit
-
Yes
Description
Rebalance test, 3 -> 4 nodes, 1 bucket x 100M x 2KB, 10K mixed ops/sec
We had a huge regression in ep-engine which was fixed recently by Mike.
But rebalance is still slower than it used to be (~20minutes vs. 50 minutes),
Master events indicate some unusual slowness in vbuckets movements. E.g., now extra time is spent in:
takeoverStarted -> takeoverEnded
vbucketStateChange (the last one)
vbucketStateChange -> vbucketMoverTerminate
This is not quite fair comparison with TAP-based rebalance but on average each individual movement is 3-4x slower now.
You can try to play with this profile:
http://cbmonitor.sc.couchbase.com/reports/movements/?filename=2b20aa8fee5d480fb85ba9e1b46fefe6
Similar diagram for TAP:
http://cbmonitor.sc.couchbase.com/reports/movements/?filename=65befd4031914ba1ae811932ce41def0