Replication queue limiting magma throughput

Build 6.5.0-3573

Observed in one of our magma tests that replication queue rises to tens of millions, which pushes magma to lower RR and limits write throughput. Write queue was not building for storage.

For couchbase we saw that for 1B docs(1K size) it goes to 0.4% RR (2nodes, 64GB memory quota)
Here are logs for run-
https://s3.amazonaws.com/bugdb/jira/replica_rr/collectinfo-2019-06-25T102930-ns_1%40172.23.97.28.zip
https://s3.amazonaws.com/bugdb/jira/replica_rr/collectinfo-2019-06-25T102930-ns_1%40172.23.97.29.zip

Turning off replication shows higher throughput for magma for lower RR.
Job- http://perf.jenkins.couchbase.com/job/rhea-dev/64
Cbmonitor link- http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=rhea_65-10000_custom_load_3e49

We tried to check with ephemeral bucket and saw replication queue(ep_dcp_replica_items_remaining) rising to ~35M while inserting 80M docs.