STREAM_REQUEST "lost" by DcpProducer during replication stream setup

This ticket is being spun out of a customer issue where we see

Disconnecting because a message has not been received for the DCP idle timeout of 360s. Received last message (e.g. mutation/noop/StreamEnd) 361s ago. DCP noop interval is 1s.

This is originated from the consumer side of the connection.

Unfortunately because the connection has been closed we do not have any detailed information on the connection at the time it stopped responding.

However it is assumed that the producer was in a paused state and was not woken-up.

In addition to the customer issue, we believe the issue may also have appeared in MB-42115. Where we also see a DCP idle - causing a disconnect.
In that case we are also disconnected from the consumer

2020-10-16T02:34:49.507127-07:00 INFO 71: (default) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.105.159->ns_1@172.23.105.206:default - Disconnecting because a message has not been received for the DCP idle timeout of 360s. Received last message (e.g. mutation/noop/StreamEnd) 361s ago. DCP noop interval is 1s.

It was originally thought that the reason was due to the consumer being OOM snd therefore not responding to the producer. However when a connection is paused then the ConnManager task should run every X seconds and send a Noop (and there’s no mem check in that area)