GoXDCR: Replication is stuck retrying to send documents

Build
-------
3.5.0-1444

First chance at manual testing

1. 2 one node clusters, enabled goxdcr and created replication. (.186 --> .188)
2. injected keys at 6k sets/sec. ./cbworkloadgen -n 10.3.4.186:8091 -r .9 -i 10000000000 --prefix=aruna. -b bucket -u Administrator -p password
I stopped after loading ~6M keys.
3. After replicating 1136111 keys, replication stopped. Workload continued for 15 mins, replication did not proceed.

Each doc is only 10bytes in size, checkpoint_interval=1800s

Attaching logs and screenshots.

On .186, looks like we've been retrying to send doc 'pymc76' and 'pymc43376' for close to 40 mins.Pipeline is however not considered broken or reconstructed.

PipelineManager17:29:18.949268 [INFO] Replication Status = map[replicationSpec/1665adc1dca3f23ef0c75d7c8f655b8b/bucket/default:name=

{replicationSpec/1665adc1dca3f23ef0c75d7c8f655b8b/bucket/default}

, status=

{Replicating}

, errors={[

{"time":"2015-03-04T16:52:15.94547928-08:00","errMsg":"map[xmem_replicationSpec/1665adc1dca3f23ef0c75d7c8f655b8b/bucket/default_10.3.4.188:11210_1:Failed to resend document pymc43376, has tried to resend it 11, maximum retry 10 reached]"}

,

{"time":"2015-03-04T16:51:03.490019777-08:00","errMsg":"map[xmem_replicationSpec/1665adc1dca3f23ef0c75d7c8f655b8b/bucket/default_10.3.4.188:11210_1:Failed to resend document pymc76, has tried to resend it 11, maximum retry 10 reached]"}

]}, progress={Pipeline is running