[system test] rebalance hang when add nodes to cluster

Cluster information:

• 8 centos 6.2 64bit server with 4 cores CPU
• Each server has 32 GB RAM and 400 GB SSD disk.
• SSD disk format ext4 on /data
• Each server has its own drive, no disk sharing with other server.
• Load 15 million items to both buckets
• Cluster has 2 buckets, default (11GB) and saslbucket (11GB) with consistent view enable. For 2 buckets, we use only 68% total RAM of system.
• Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)

• Create cluster with 4 nodes installed couchbase server 2.0.0-1746

10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40

• Data path /data
• View path /data

• Add 4 nodes to cluster and rebalance
  10.6.2.42
  10.6.2.43
  10.6.2.44
  10.6.2.45

• rebalance hang

• Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-hang-20120920.tgz

Link to atop file of all nodes https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-8nodes-1746-reb-hang-20120920.tgz