System testing for XDCR: XDC Replication queue is continuosly growing under moderate load.

1.Setup
Unidirectional replication from cluster1, default --> cluster2, default
Unidirectional replication from cluster2, sbucket--> cluster1, sbucket

2.Setup a 4K front end load on default [create, update, delete]
Setup a 10K front end load on sbucket [create, update, delete, expires]

3. Load 40M items on default bucket
Load 200M items on sbucket

4. Cluster running up for 48 hours+

5. XDC Replication queue is growing on both the buckets.

Seeing these errors on replication.

from default replication -
Error replicating vbucket 535: {http_request_failed, "POST", "http://Administrator:*****@ec2-175-41-185-71.ap-southeast-1.compute.amazonaws.com:8092/sbucket%2f535%3bc8731525718bcbdd0bf0382e420c453f/_revs_diff", {error,

{error,timeout}}}


From sbucket replication -
Error replicating vbucket 535: {http_request_failed, "POST", "http://Administrator:*****@ec2-175-41-185-71.ap-southeast-1.compute.amazonaws.com:8092/sbucket%2f535%3bc8731525718bcbdd0bf0382e420c453f/_revs_diff", {error,{error,timeout}

}}

Live clusters here :
http://ec2-50-18-140-172.us-west-1.compute.amazonaws.com:8091/index.html#sec=replications
http://ec2-54-251-5-97.ap-southeast-1.compute.amazonaws.com:8091/index.html#sec=replications

• Stopped the front end load for now.