Details
Description
- 2 node cluster
- 2 buckets
- Bucket 'bkt' had a very high percentage of sets in its front end load.
- Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
- Rebalance froze at around 98%.
- Stopped front end loads, disk write queue drained.
- Mem used for both nodes, greater than higher water mark.
- Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
- Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
Cluster diags:
1 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz
Attached the cbstats of all, raw memory for both nodes.