[system test]query node becomes unresponsive after cluster KV topology changes.( rebalancing)

1. load data + indexes
2. Rebalance In out data nodes
3. Keep incoming mutations on the cluster.

Seen this on 2 instances/runs so far, I get no result from one of the query nodes.

Issuing a kill -3 interrupt, query service restart solves the problem.

core : https://s3.amazonaws.com/bugdb/core.tar.gz

tail -f query.log shows couple of connection refused to the index nodes.

2015-07-17T14:52:49.081Z-07:00 [Error] WatcherServer.runOnce() error : dial tcp 10.6.2.238:9100: connection refused
2015-07-17T14:52:50.435Z-07:00 [Error] WatcherServer.runOnce() error : dial tcp 10.6.2.234:9100: connection refused
2015-07-17T14:52:50.992Z-07:00 [Error] WatcherServer.runOnce() error : dial tcp 10.6.2.237:9100: connection refused
2015-07-17T14:52:51.082Z-07:00 [Error] WatcherServer.runOnce() error : dial tcp 10.6.2.238:9100: connection refused
2015-07-17T14:52:56.41Z-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification
2015-07-17T14:52:56.757Z-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
2015-07-17T14:52:56.758Z-07:00 [Info] Refreshed Indexer List: [10.6.2.234:9100 10.6.2.237:9100 10.6.2.238:9100]
2015-07-17T14:53:11.43Z-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification
2015-07-17T14:53:11.796Z-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh.
2015-07-17T14:53:11.797Z-07:00 [Info] Refreshed Indexer List: [10.6.2.234:9100 10.6.2.237:9100 10.6.2.238:9100]