rebalance-in fails to wait for bucket deletion after graceful failover.

Test first loads 100M documents and did a graceful failover. It was fine.

Test then add back the node (.14) and starts rebalance. It didn't complete.

(If I then manually trigger rebalance again, it is fine.)

(Also, if run with 10M documents total, test also passes.)

(The 100M case is very reproducible on Ares.)

REST call to pools/default/tasks got this:

{u'status': u'notRunning', u'statusIsStale': False, u'errorMessage': u'Reba lance failed. See logs for detailed reason. You can try rebalance again.', u'type': u'rebalance', u'masterRequestTimedOu t': False}

Here is some log snippet from the console:

Failed to wait deletion of some buckets on some nodes: [{'ns_1@172.23.96.14',
{'EXIT',

{old_buckets_shutdown_wait_failed, ["bucket-1"]}

}}]

Here is something possibly relevant in the ns_server.debug.log on the .14 node:

[ns_server:error,2015-08-25T00:47:09.552-07:00,ns_1@172.23.96.14:timeout_diag_logger<0.129.0>:timeout_diag_logger:do_diag:105]Got timeout {slow_bucket_stop,{{single_bucket_kv_sup,"bucket-1"},
<0.369.0>,supervisor,
[single_bucket_kv_sup]}}