XDCR: Items not being replicated to non-master nodes on the destination

With any of the automation tests, for e.g. :
./testrunner -i temp.ini -t xdcr.uniXDCR.unidirectional.load_with_ops,items=10000,rdirection=unidirection,ctopology=chain
./testrunner -i temp.ini -t xdcr.uniXDCR.unidirectional.async_load_with_ops,items=10000,rdirection=unidirection,ctopology=chain

where temp.ini is a 2:2 cluster set up

{ source = [10.3.2.94, 10.3.2.95], destination = [10.3.2.96, 10.3.2.97]}

It is seen that replication is set up as expected, but when load is started replication on the destination side happens at just one of the 2 nodes, causing only 50% of the items showing up on the destination.

Also, when tried with 3:3 cluster set up, items not being replicated to 2 nodes on the destination. => ((1/no_of_desination_nodes)*items) being replicated just to the master node at the destination cluster.

Seen crash report:

• • Reason for termination ==
  • {badmatch,
    Unknown macro: {error,all_nodes_failed, <<"Failed to grab remote bucket info from any of known nodes">>}

    }

[ns_server:debug,2012-09-26T11:50:42.224,ns_1@10.3.2.95:<0.8120.0>:remote_clusters_info:maybe_extract_important_buckets_props:789]skipping node without couchApiBase: [{<<"systemStats">>,
{struct,
[

{<<"cpu_utilization_rate">>,0.0}

,

{<<"swap_total">>,6840901632}

,

{<<"swap_used">>,0}

]}},
{<<"interestingStats">>,
{struct,
[

{<<"couch_docs_actual_disk_size">>, 8442702}

,

{<<"couch_views_actual_disk_size">>,0}

,

{<<"curr_items">>,0}

,

{<<"curr_items_tot">>,0}

,

{<<"mem_used">>,31116760}

,

{<<"vb_replica_curr_items">>,0}

]}},

{<<"uptime">>,<<"145">>}

,

{<<"memoryTotal">>,4157632512}

,

{<<"memoryFree">>,2901553152}

,

{<<"mcdMemoryReserved">>,3172}

,

{<<"mcdMemoryAllocated">>,3172}

,

{<<"replication">>,1.0}

,

{<<"clusterMembership">>,<<"active">>}

,

{<<"status">>,<<"healthy">>}

,

{<<"hostname">>,<<"10.3.2.97:8091">>}

,

{<<"clusterCompatibility">>,131072}

,

{<<"version">>, <<"2.0.0-1768-rel-enterprise">>}

,

{<<"os">>,<<"x86_64-unknown-linux-gnu">>}

,
{<<"ports">>,
{struct,
[

{<<"proxy">>,11211}

,

{<<"direct">>,11210}

]}}]
[ns_server:error,2012-09-26T11:50:42.225,ns_1@10.3.2.95:<0.8120.0>:remote_clusters_info:do_build_ix_to_couch_uri_dict:904]Was not able to find node corresponding to server 10.3.2.97:11210
[error_logger:error,2012-09-26T11:50:42.225,ns_1@10.3.2.95:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: xdc_vbucket_rep:init/1
pid: <0.7775.0>
registered_name: []
exception exit: {badmatch,{error,all_nodes_failed,
<<"Failed to grab remote bucket info from any of known nodes">>}}
in function gen_server:terminate/6
ancestors: [<0.7479.0>,<0.7474.0>,xdc_replication_sup,ns_server_sup,
ns_server_cluster_sup,<0.60.0>]
messages: []
links: [<0.7479.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 17711
stack_size: 24
reductions: 11656
neighbours: