replicateTo with multiple replicas not guaranteed to target promoted replica

The expected behavior of replicateTo is that the write is durable i.e. it will survive if the node with the active vbucket becomes unavailable.

However, that is not the case today.

If user chooses replicateTo to be less than total number of replicas, they can 'lose' writes that were acked as being replicated. We do not choose the "most up to date" replica when doing a failover and hence a DCP stream rollback triggers a reverse of the write from the node(s) that acked it.