rebalance froze when node failed over and added back (observed mem used > high water mark for bucket)

• 2 node cluster
• 2 buckets
• Bucket 'bkt' had a very high percentage of sets in its front end load.
• Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
• Rebalance froze at around 98%.
• Stopped front end loads, disk write queue drained.
• Mem used for both nodes, greater than higher water mark.
• Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
• Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

Cluster diags:
1 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

2 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

Attached the cbstats of all, raw memory for both nodes.