rebalance hangs on long-running rebalance test

In osx dev environment, setup 4 nodes with the gen-cluster-scripts and keep rebalancing them (add, remove, rebalance, repeat), while under memcachetest load. I did this via the automated ns_server/test/Makefile (which is currenty pushed to gerrit)...

cd ns_server
(cd test && make prep)
./cluster_run.sh

In another terminal...

cd test
make test-rebalance-loop

In another terminal...

./memcachetest -h 127.0.0.1:12001 -i 1000 -c 10000 -l

Eventually, one of the rebalance requests (through the membase-cli) will hang. In this case, the 1065th rebalance attempt...

------------------------- 265 - Sat Nov 6 13:16:45 PDT 2010
make joiners-do NODES=4 DO=server-remove
../../membase-cli/membase rebalance -c 127.0.0.1:9000 \
-u Administrator -p 111111 \
--server-remove=127.0.0.1:9001
INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (thousands of dots follow)

After that, I manually stopped the rebalance via the web UI, and will be attaching diag soon.