{UPR}:: Rebalance-out failing due to bad replicators

Tested on 1011, 1005, both ubuntu and centos are seeing this issue

1. Create a 7 node cluster
2. Create a default bucket
3. Add 100 K items
4. Rebalance-out 1 Node (10.3.121.69)
5. Do Ops for Gets

Step 4 and Step 5 act in parallel.

Rabalance exits with the following error::

Bad replicators after rebalance:
Missing = [

{'ns_1@10.3.121.63','ns_1@10.3.121.64',0}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.64',1}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.64',2}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.64',3}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.64',56}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.66',4}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.66',5}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.66',6}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.66',57}

,

{'ns_1@10.3.121.63','ns_1@10.3.121.66',58}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.63',19}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.63',20}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.63',21}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.63',22}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.63',59}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.66',23}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.66',24}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.66',25}

,

{'ns_1@10.3.121.64','ns_1@10.3.121.66',60}

,

{'ns_1@10.3.121.64','ns_1@10.5.2.13',26}

,

{'ns_1@10.3.121.64','ns_1@10.5.2.13',29}

,

{'ns_1@10.3.121.64','ns_1@10.5.2.13',30}

,

{'ns_1@10.3.121.64','ns_1@10.5.2.13',31}

,

{'ns_1@10.3.121.64','ns_1@10.5.2.13',61}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.63',38}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.63',39}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.63',40}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.63',62}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.64',41}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.64',42}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.64',43}

,

{'ns_1@10.3.121.66','ns_1@10.3.121.64',63}

,

{'ns_1@10.3.121.66','ns_1@10.5.2.13',44}

,

{'ns_1@10.3.121.66','ns_1@10.5.2.13',47}

,

{'ns_1@10.3.121.66','ns_1@10.5.2.13',48}

,

{'ns_1@10.3.121.66','ns_1@10.5.2.13',49}

,

{'ns_1@10.3.121.66','ns_1@10.5.2.13',64}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.63',65}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.63',74}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.63',75}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.63',76}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.64',66}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.64',77}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.64',78}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.64',79}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.66',67}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.66',80}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.66',81}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.66',82}

,

{'ns_1@10.5.2.13','ns_1@10.3.121.66',83}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.63',68}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.63',92}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.63',93}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.63',94}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.64',69}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.64',95}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.64',96}

,

{'ns_1@10.5.2.14','ns_1@10.3.121.64',97}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.63',71}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.63',110}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.63',111}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.63',112}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.64',72}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.64',113}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.64',114}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.64',115}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.66',73}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.66',116}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.66',117}

,

{'ns_1@10.5.2.15','ns_1@10.3.121.66',118}

]
Extras = []

Test Case:: ./testrunner -i centos.ini get-cbcollect-info=False,get-logs=False,stop-on-failure=False,get-coredumps=True,force_kill_memached=False,verify_unacked_bytes=True,dgm=True,total_vbuckets=128,std_vbuckets=5 -t rebalance.rebalanceout.RebalanceOutTests.rebalance_out_get_random_key,nodes_out=1,items=100000,value_size=256,skip_cleanup=True,GROUP=OUT;BASIC;P0;FROM_2_0

Will attach logs asap