Rebalance hanging after trying to "swap" all nodes in a cluster

Description From Perry Krug 2010-11-01 19:11:50 [reply] Private

Here's what I did:
-Setup a 2 node cluster (10.1.5.7, 10.1.5.8)
-Run this command line to add 2 more nodes (10.1.5.9, 10.1.3.100) and remove
the original 2 at the same time:
/opt/membase/bin/cli/membase rebalance -c 10.1.5.7 -u Administrator -p norths
--server-add=10.1.5.9 --server-add=10.1.3.100 --server-remove=10.1.5.7
--server-remove=10.1.5.8

This says it completed successfully but when I went into 10.1.5.9 it showed
that 10.1.5.8 was still on the pending rebalance list. I clicked rebalance via
the UI and it's now stuck at 0%.

Diag from the three nodes stuck in rebalancing attached.

------- Comment #1 From Perry Krug 2010-11-01 19:12:14 [reply] ------- Private

Created an attachment (id=863) [details]
diag1

------- Comment #2 From Perry Krug 2010-11-01 19:12:30 [reply] ------- Private

Created an attachment (id=864) [details]
diag2

------- Comment #3 From Perry Krug 2010-11-01 19:12:43 [reply] ------- Private

Created an attachment (id=865) [details]
diag3