Could you give a bit more background on this issue? You've marked it as a bug, but this just sounds like an observation that swap rebalance takes longer than a single rebalance in?

As such, that doesn't sound like a bug to me (possibly an improvement?) - unless it's a regress from some previous build.

(As an aside, swap-rebalance is still quicker than node_in + node_out, so not clear this is even unexpected behaviour).

If you think this is a bug, please update to include expected behaviour and actual behaviour. If not then change to an improvement making clear what you think should be improved.

Dave Rigby this is more of an investigation at this point i.e. why is the swap rebalance time so much more than rebalance-in? Is there a legitimate explanation for this? Or is it odd behavior?

This has been observed on the high bucket density testing with 30 buckets where rebalance time is already high. So swap rebalance increasing by 50% really spikes the rebalance time.

If you think it should first be investigated by ns_server team, please re-assign.

I am changing it to improvement and re-assigning to you so that you can suggest next steps.

ns_server orchestrates rebalances, including when vBuckets are moved. I’d suggest any analysis starts there. Reassigning.

I have the other high bucket density rebalance ticket MB-32645

So, assigning this to me as well.

setting initial fix version to Mad Hatter so that investigation occurs in MH timeframe.  Please update fix version once investigation completes.

Different types of rebalances can take different amount of time as they may be moving different # of vbuckets.
 
The description does not say how many nodes were involved in the swap, rebalance-in and rebalance-out. That will tell us how many vBuckets are being moved during each.
 
Also, the links to the logs are not working.
 
E.g. Check out the weekly results from the tests run by perf team below.
 
4 > 4 Swap  took 63.9 mins where as rebalance-in (4>5) took 37.2 and rebalance out (5->4) took 34.4. 
 
swap took ~70% more time than the above rebalance-in and rebalance-out.
 
4->4 swap rebalance moves 512 vBuckets where as 4->5 rebalance-in and 5->4 rebalance out move only 408 vBuckets each. 
 
http://showfast.sc.couchbase.com/#/timeline/Linux/reb/kv/DGM 
 
Please specify the # of nodes involved during each rebalance.

Poonam Dhavale , updated description with #nodes involved.

3 -> 3 swap rebalance moves 680 vBuckets whereas rebalance in(3 -> 4) and  rebalance out(4 -> 3) each move 512 vBuckets.

So, it is expected 3 -> 3 swap to take longer than rebalance in(3 -> 4) and  rebalance out(4 -> 3).

 Is it expected to that swap will always take 50-60% longer? No, that depends on various other factors such as amount of data in the bucket, the load on the system, cpu and other resources available etc.

In the showfast link I had posted earlier, swap took ~70% longer. That was for DGM.

Here is the showfast link for rebalance sanity tests run by perf team. swap takes around ~41 - 55% longer.

http://showfast.sc.couchbase.com/#/timeline/Linux/reb/kv/Sanity 

Given that the observations in this ticket (swap rebalance time is ~50-60% higher) are within the range seen during the weekly perf tests, I think, we can close this ticket.

Thanks Poonam for investigating this issue. 

It still is cause for concern that moving 25% more vbuckets takes 50% more time. I understand that this is what we see in the weekly perf tests as well. Feels like there is room for improvement here.

We don't have to commit to fixing it immediately but can we investigate further to nail down the root cause?

Hi Shivani,

In the 3->3 swap case described above, it is moving 33% more vBuckets. 

Regarding why it is taking 50-60% more time when it is moving only 33% more vBuckets:

In addition to moving higher #of vBuckets, swap also has different rebalance characteristic when compared to reb-in/reb-out. This affects the vBucket scheduling logic which also plays a role in how fast a rebalance can go. 

The vBucket scheduling logic (described in the link below) allows limited # of backfills and moves for nodes that are acting as the old or the new master.

Consider a 3 node cluster, N0, N1, N2.

• 3->3 swap rebalance to remove N2 and replace it with N3.
  • 341 active vBuckets will move from N2 to N3. N2 is the old master for all of these.
  • 341 replica vBuckets will move, the master for these is one of N0 or N1.
• 3->4 rebalance in to add N3:
  • 256 active vBuckets will move to N3. The master for these is one of N0, N1, N2.
  • 256 replica vBuckets will move to N3. The master for these is one of N0, N1, N2.
• 4 -> 3 rebalance out to remove N3 will have similar characteristics as described above for 3 ->4 rebalance in.

So, in above swap rebalance, one node (N2) is the old master for majority of the vBuckets (341). 

Whereas for reb-in & reb-out, the current/old master for vBucket movements are more or less evenly distributed across the 3 nodes. (170 each).

This affects the order in which vBuckets are moved and how many are moved at a time.

But, I have added a note to the design doc below to investigate whether we can improve on swap rebalance time. This will be for Cheshire Cat.

https://docs.google.com/document/d/1pqNY7GufVCyiEk8ikkltyCu-15KtqZKYlzpNSKiV2mI/edit#heading=h.4iy0vndbwik5 

Thanks Poonam for patiently explaining in detail. It makes more sense now why swap takes longer.

Since you are already tracking investigating Cheshire Cat improvement in your document, we can close this ticket.