Please see details explanation in
Started over with a pair of single node clusters (10.1.3.71 and 10.1.3.72), threads=4, expiration=10
– Added a node (10.1.3.235) on to the source cluster, rebalance.
- Fall in destination ops-per-sec towards the end of rebalance on the source`s side.
- But picks up once rebalance actually finishes.
- Failover a node on source cluster
- Couldn`t add it back and rebalance for some reason, so removed it completely, rebalanced and then added another node again, but rebalance is now stuck (between 10.1.3.71 and 10.1.3.235)
Basically, memcached timed out on numerous request(s) to deregister replica building tap from ns_server. Thus post-rebalance clean up indefinitely got stuck.