replica streaming triggers: ActiveStream::markDiskSnapshot:sent snapshot marker to client with snap start <= previous snap end

Following exception can be triggered when streaming from a replica vbucket

CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveStream::markDiskSnapshot:sent snapshot marker to client with snap start <= previous snap end vb:280 lastSentSnapStart:21402 lastSentSnapEnd:21469 snapStart:21402 snapEnd:27274 sid:sid:none producer name:eq_dcpq:dcpdrain-82249_0 lastReadSeqno:21401 curChkSeqno:27237

The following sequence triggers.

• Stream begins from 0
• Backfill triggered, e.g. seqno 1 to 100 from backfill, but during markDiskSnapshot we extend the snapshot into memory (merge)
  • https://src.couchbase.org/source/xref/7.6.2/kv_engine/engines/ep/src/dcp/active_stream.cc?r=678075eb#372-389
  • e.g. snapshot sent as 1 to 150 (disk is 1 to 100, memory is 101 to 150).
• backfill transitions to in-memory, trying to send 101 to 150.
• But cursor dropped at say 120.
• next backfill scheduled for 120 to 150
• markDiskSnapshot now tries to send a snapshot 120 to 150 and we hit the exception.

This occurs because from the logic the second snapshot is not following the first.

E.g. we've sent

• snapshot 1 to 150
• snapshot 120 to 150

At this point i don't believe this to be a wider protocol bug, I don't believe DCP guarantees it would reach the end of the snapshot before starting a new one, e.g. both the following are valid.

• snapshot 1 to 150
• snapshot 151 to 160

and

• snapshot 1 to 150
• snapshot 120 to 150