XDCR: Seeing very slow ( currently 0-4 items) replication to destination cluster on enabling replication from source node.

1.Setup a replication between 2 clusters ( 3 node: 3node)
2. Load 100k items on the source

Expect 100k items to be replicated to destination-
Output
1. Seeing spike of data replicated ( stats are choppy)
After 10 minutes, seeing 0 items on the destination cluster

Every 2.0s: /opt/couchbase/bin/cbstats localhost:11210 all -b bucket-0 | grep curr_items Fri Jul 6 11:26:38 2012

curr_items: 0
curr_items_tot: 0
vb_active_curr_items: 0
vb_pending_curr_items: 0
vb_replica_curr_items: 0

These tests ran correct until the 1409 build on Jenkins

--------
Hi Junyi,

For the XDCR Jenkins run on build 1410+, we are seeing few failures with "Replication Failed"

I am monitoring the current run ./testrunner -i /tmp/centos-64-6node-xdcr.ini get-logs=True -t xdcr.XDCRTests.test_continuous_unidirectional_sets,num_items=100000,poll_sleep=5,poll_timeout=300

And I see that the source cluster loads 100k Items, but no items are replicated across to the destination cluster.

Seeing a number of gets being performed. Attaching some screenshots
10.1.3.96 - Source
10.1.3.93 - Destination

The same test ran successfully on my own VMs ( exactly same configuration). Any idea on how to figure out why we see only Gets/ and No sets on the source cluster?

1. SOUCRE cluster
2. DESTINATIOn cluster

Attaching screenshots from source and destination nodes