{UPR} :: Incremental Rebalance-in [hangs/stuck] with ops

Version 3.0.0-730, vbuckets=128

Nodes: 10.6.2.144-150

1. Add 3 nodes
2. Create default bucket
3. Add 100K Items
4. Rebalance in 2 nodes with ops in parallel
5. Rebalance in 2 nodes with ops in parallel

Step 5 observes Rebalance failing :: being stuck/hangs

http://qa.hq.northscale.net/view/3.0.0/job/centos_x64--02_04--Rebalance_tests_UPR-P0/148/console

logs attached

Rebalance Steps which led to failure

2014-05-23 21:22:59 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.6.2.149:8091 to this cluster @10.6.2.144:8091
2014-05-23 21:22:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.6.2.148:11210 default
2014-05-23 21:22:59 | INFO | MainProcess | test_thread | [rest_client.get_nodes] Node 10.6.2.149 not part of cluster inactiveAdded
2014-05-23 21:22:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.6.2.146:11210 default

2014-05-23 21:23:02 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.6.2.150:8091 to cluster
2014-05-23 21:23:02 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.6.2.150:8091 to this cluster @10.6.2.144:8091

2014-05-23 21:23:15 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.6.2.146%2Cns_1%4010.6.2.144%2Cns_1%4010.6.2.145%2Cns_1%4010.6.2.147%2Cns_1%4010.6.2.148%2Cns_1%4010.6.2.150%2Cns_1%4010.6.2.149