A couple of our regression tests started intermittently failing recently because of some missing documents that are supposed to be streamed via DCP within a certain timeout (60 seconds).

The tests use cluster_run with 3 nodes as follows:
Node n0: data + analytics
Node n1: data
Node n2: analytics

At around time 2020-06-14T19:52:40.348-07:00, the beer-sample bucket is created and loaded. After that, Analytics starts DCP streams to the bucket.
When I checked the logs, I noticed that we are receiving all the documents from one of the data nodes but none from the other.

Following the memcached logs for a vbucket from each node, we can see the following:

Node n0 (vb:1):

Node: n1 (vb:516) the backfill snapshot is never sent:

We haven't made any major changes to the DCP streaming client and that test used to pass all the time within that configured timeout of 60 seconds.

cbcollect_info from the nodes is attached.