[Ephemeral] Collection backfill does not allow collection highSeqno <= purgeSeqno on reconnect

There is a collections-related feature under MB-32358, which allows a stream to re-connect when the requested startSeqno is behind the purgeSeqno, as long as the stream is up-to-date on the collection.

Unfortunately, we implemented that logic for persistent buckets only, but not for ephemeral buckets.

The result is that the streamRequest matching the conditions of MB-32358 succeeeds, but the DCP stream is immediately interrupted and closed with Status=Rollback shortly after that.

The Rollback in the stream end doesn't contain the rollbackSeqno, it is just informational. Clients just retry the stream request, expecting that if there is a need for rollback, that will be signalled when the stream is first created (as a response status code from the STREAM_REQ command).

This can lead to an endless loop of a) stream created b) stream closes with rollback.

Note: As of 7.2.3, KV supports the DCP_ADD_STREAM_FLAG_IGNORE_PURGED_TOMBSTONES flag, which avoids rollback in this case. Services which pass this flag are immune to the endless loop. For example, Eventing started using this flag in MB-60453 in 7.2.5, so this bug cannot affect the Eventing service in 7.2.5+.

While the initial stream starting a seqno 0 should be successful, when a service then reconnects at a seqno higher than 0 while the collection filtered by the stream has a collection highSeqno < purgeSeqno, this bug is hit.

Workaround

As a temporary workaround, it is possible to write or update a document into the collection filtered by the stream, to increase the collection highSeqno past the purgeSeqno. This could be a GAT (Get and Touch) on an existing document in the collection, or a new document created with a 1 second expiry time. This will move the collection highSeqno > purgeSeqno.

When the client attempts to create the stream, it will be told to rollback to seqno 0 and catch up with all changes successfully.