quickly shutting down a node after creating a large cluster causes server menelaus server error

In this test I iteratively created a cluster of 8 node by starting with one node and adding one node at a time. Once all nodes were created, I logged into the web console of one of the nodes and confirmed that all 8 nodes were available. I then went to the 8th node (ec2-174-129-158-21.compute-1.amazonaws.com) and shutdown the Northscale Server. To shutdown the server I executed the following

/etc/init.d/northscale stop

At which point I then went back to the web console and noticed that the Manage -> Servers page appeared to be attempting to load, eventually it presented me with the following error

http://skitch.com/eric.d.lambert/dje2n/safari

The web console for all nodes in the cluster present the same issue.

The log shows the following error

Server error during processing: ["web request failed",

{path,"/pools/default"}

,

{type,exit}

, {what, {timeout, {gen_server,call, [

{global,'ns_orchestrator-default'}

, rebalance_progress]}}}, {trace, [

{gen_server,call,2}

,

{menelaus_web,build_pool_info,4}

,

{menelaus_web,handle_pool_info,2}

,

{menelaus_web,loop,3}

,

{mochiweb_http,headers,5}

,

{proc_lib,init_p_do_apply,3}

]}] (repeated 5 times)

It may be worth noting that vbucketmigrator was running on the 8th node when I shut it down

100 2635 1 0 Jun19 ? 00:00:00 /opt/NorthScale/1.6.0a/erlang-13b03/lib/erlang/erts-5.7.4/bin/epmd -daemon
postfix 22919 2211 0 04:34 ? 00:00:00 pickup -l -t fifo -u
100 23106 1 0 05:25 ? 00:00:01 /opt/NorthScale/1.6.0a/erlang-13b03/lib/erlang/erts-5.7.4/bin/beam.smp – -root /opt/NorthScale/1.6.0a/erlang-13b03/lib/erlang -progname erl – -home /opt/NorthScale – -pa /opt/NorthScale/1.6
100 23123 23106 0 05:25 ? 00:00:00 sh -s disksup
100 23124 23106 0 05:25 ? 00:00:00 /opt/NorthScale/1.6.0a/erlang-13b03/lib/erlang/lib/os_mon-2.2.4/priv/bin/memsup
100 23126 23106 0 05:25 ? 00:00:00 /opt/NorthScale/1.6.0a/erlang-13b03/lib/erlang/lib/os_mon-2.2.4/priv/bin/cpu_sup
100 23174 23106 0 05:25 ? 00:00:00 inet_gethost 4
100 23175 23174 0 05:25 ? 00:00:00 inet_gethost 4
100 23182 23106 0 05:27 ? 00:00:00 ./bin/moxi/moxi -Z port_listen=11211,downstream_max=1 -z auth=,url=http://127.0.0.1:8080/pools/default/bucketsStreamingConfig/default,#@ -p 0
100 23192 23106 0 05:27 ? 00:00:00 ./bin/memcached/memcached -p 11210 -X ./bin/memcached/stdin_term_handler.so -E ./bin/ep_engine/ep.so -B binary -r -e vb0=false;ht_size=12289;ht_locks=23;dbname=/opt/NorthScale/1.6.0a/data/ns_1
100 23220 23106 0 05:27 ? 00:00:00 ./bin/vbucketmigrator/vbucketmigrator -h 10.253.47.95:11210 -d 10.253.25.235:11210 -v -b 180 -b 177
100 23232 23106 0 05:27 ? 00:00:00 ./bin/vbucketmigrator/vbucketmigrator -h 10.253.47.95:11210 -d 10.253.85.204:11210 -v -b 179 -b 178
100 23248 23106 0 05:27 ? 00:00:00 ./bin/vbucketmigrator/vbucketmigrator -h 10.253.47.95:11210 -d 10.253.11.144:11210 -v -b 181 -b 176
100 23251 23106 0 05:27 ? 00:00:00 ./bin/vbucketmigrator/vbucketmigrator -h 10.253.47.95:11210 -d 10.253.5.156:11210 -v -b 174 -b 170 -b 87 -b 47 -b 42
100 23254 23106 0 05:27 ? 00:00:00 ./bin/vbucketmigrator/vbucketmigrator -h 10.253.47.95:11210 -d 10.253.66.191:11210 -v -b 183 -b 175 -b 91 -b 88 -b 46 -b 43 -b 12
100 23257 23106 0 05:27 ? 00:00:00 ./bin/vbucketmigrator/vbucketmigrator -h 10.253.47.95:11210 -d 10.253.74.32:11210 -v -b 173 -b 172 -b 90 -b 89 -b 86 -b 85 -b 45 -b 44 -b 21 -b 11 -b 10 -b 9 -b 8 -b 4