cbrecovery doesn't work, it always stuck

all tests failed with the same issue: cbrecovery stuck http://qa.sc.couchbase.com/job/ubuntu_x64--38_01--cbrecovery-P1/51/

for example, we failover 2 from 4 nodes on source and run cbrecovery

test logs:

2015-01-21 07:30:05 | INFO | MainProcess | test_thread | [cbRecoverytests.cbrecover_multiple_failover_swapout_reb_routine] Failing over 2 nodes on source ..
2015-01-21 07:30:06 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.107.182:8091 with graceful=False
2015-01-21 07:30:06 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.107.182 successful
2015-01-21 07:30:07 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.107.181:8091 with graceful=False
2015-01-21 07:30:07 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.107.181 successful
2015-01-21 07:30:07 | INFO | MainProcess | Cluster_Thread | [task.execute] 0 seconds sleep after failover, for nodes to go pending....
2015-01-21 07:30:07 | INFO | MainProcess | test_thread | [rest_client.add_node] adding remote node @172.23.107.103:8091 to this cluster @172.23.107.180:8091
2015-01-21 07:30:17 | INFO | MainProcess | test_thread | [rest_client.add_node] adding remote node @172.23.107.105:8091 to this cluster @172.23.107.180:8091
2015-01-21 07:30:22 | INFO | MainProcess | test_thread | [xdcrbasetests.sleep] sleep for 15 secs. ...
2015-01-21 07:30:38 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.244 with username : root password : couchbase ssh_key:
2015-01-21 07:30:39 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.244
2015-01-21 07:30:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.244: sudo cat /proc/cpuinfo
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.244: df -Th
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.244: sudo cat /proc/meminfo
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.244: hostname
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-01-21 07:30:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.244: hostname -d
2015-01-21 07:30:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully
2015-01-21 07:30:43 | INFO | MainProcess | Cluster_Thread | [task.execute] command was executed: '/opt/couchbase/bin/cbrecovery http://172.23.106.244:8091 http://172.23.107.180:8091 -b default -B default -u Administrator -p password -U Administrator -P password '
2015-01-21 07:31:03 | INFO | MainProcess | Cluster_Thread | [task.check] cbrecovery progress: {u'code': u'ok', u'uuid': u'4ca79b76647d0a4d476fc1726a25c7db', u'recoveryMap': [

{u'node': u'ns_1@172.23.107.103', u'vbuckets': [513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 680, 681, 682, 854, 855, 856, 857, 858, 859, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 890, 891, 892, 893, 894, 895, 896, 897, 898, 899, 900, 901, 902, 903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023]}

]}
2015-01-21 07:31:23 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:31:44 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:32:04 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:32:25 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:32:45 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:33:06 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:33:26 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed
2015-01-21 07:33:47 | WARNING | MainProcess | Cluster_Thread | [task.check] cbrecovery progress was not changed

...

curl -0 http://Administrator:password@172.23.107.180:8091/pools/default/buckets/default/recoveryStatus?recovery_uuid=4ca79b76647d0a4d476fc1726a25c7db
{"uuid":"4ca79b76647d0a4d476fc1726a25c7db","code":"ok","recoveryMap":[

{"node":"ns_1@172.23.107.103","vbuckets":[513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999,1000,1001,1002,1003,1004,1005,1006,1007,1008,1009,1010,1011,1012,1013,1014,1015,1016,1017,1018,1019,1020,1021,1022,1023]}

]}