Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
7.6.0, Morpheus, 7.6.2, 7.6.1, 7.6.4
-
None
-
Untriaged
-
0
-
Unknown
Description
Following exception can be triggered when streaming from a replica vbucket
CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveStream::markDiskSnapshot:sent snapshot marker to client with snap start <= previous snap end vb:280 lastSentSnapStart:21402 lastSentSnapEnd:21469 snapStart:21402 snapEnd:27274 sid:sid:none producer name:eq_dcpq:dcpdrain-82249_0 lastReadSeqno:21401 curChkSeqno:27237
|
The following sequence triggers.
- Stream begins from 0
- Backfill triggered, e.g. seqno 1 to 100 from backfill, but during markDiskSnapshot we extend the snapshot into memory (merge)
- https://src.couchbase.org/source/xref/7.6.2/kv_engine/engines/ep/src/dcp/active_stream.cc?r=678075eb#372-389
- e.g. snapshot sent as 1 to 150 (disk is 1 to 100, memory is 101 to 150).
- backfill transitions to in-memory, trying to send 101 to 150.
- But cursor dropped at say 120.
- next backfill scheduled for 120 to 150
- markDiskSnapshot now tries to send a snapshot 120 to 150 and we hit the exception.
This occurs because from the logic the second snapshot is not following the first.
E.g. we've sent
- snapshot 1 to 150
- snapshot 120 to 150
At this point i don't believe this to be a wider protocol bug, I don't believe DCP guarantees it would reach the end of the snapshot before starting a new one, e.g. both the following are valid.
- snapshot 1 to 150
- snapshot 151 to 160
and
- snapshot 1 to 150
- snapshot 120 to 150