[system test] Rebalance exited with reason timeout waiting for backfill determination

Cluster ip is 172.23.105.23
1. create 8 nodes cluster, each node has 12G RAM, HHD
2. create 2 buckets default and saslbucket, with memory quota 6G and 4G
3. Run the KV use case for 1 day:
loading 35M items to each bucket, access the data 4k ops/sec with 5% create, 5% delete, 5%expire, 5% update, 80 gets for several hours. Then with the same work load, run some rebalance and failover operations. Works good.
4 Continue the workload for another day, then try to rebalance in one node, rebalance exit with time out.

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.7961.56>,
{{badmatch,
[{'EXIT',
{timeout,

{gen_server,call, [<12869.26294.2>,had_backfill,30000]}}}]},
[{ns_single_vbucket_mover, '-wait_backfill_determination/1-fun-1-', 1}]}}}
ns_orchestrator002 ns_1@172.23.105.23 14:36:20 - Fri May 10, 2013
<0.7953.56> exited with {unexpected_exit,
{'EXIT',<0.7961.56>,
{{badmatch,
[{'EXIT',
{timeout,{gen_server,call, [<12869.26294.2>,had_backfill,30000]}

}}]},
[

{ns_single_vbucket_mover, '-wait_backfill_determination/1-fun-1-',1}

]}}}

The link for diags is https://s3.amazonaws.com/bugdb/jira/MB-8246/ns-diag-20130510162640.txt.zip