Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.2, 6.6.5, 7.0.3, 7.1.0
-
Triaged
-
1
-
No
-
KV May 22
Description
Found while working on MB-51689. This issue is similar to MB-51606 in terms of impact/workaround.
This issue is only applicable to memory snapshots as the consumer will ack the snapshot end of fully persisted disk snapshots.
Consider a replica receiving a memory snapshot as follows:
[1:Prepare(keyA), 2:Mutation(keyB)]
|
This snapshot is dealt with by the replica by three DCP messages:
- SnapshotMarker 1-2 with flag Memory (and perhaps Checkpoint but that's orthogonal to this issue)
- DcpPrepare for keyA with seqno1
- DcpMutation for keyB with seqno2
After processing 2 or 3 the flusher may run. In this scenario the flusher runs after processing 2 so the Checkpoint state is as follows:
[1:Prepare(keyA) (HPS = 1, SnapshotStart = 1, SnapshotEnd = 2)]
|
The flusher sees that this item is not the end of a snapshot range and so it does not attempt to persist the HPS value of the Checkpoint. The node then restarts. The HPS value loaded into memory is 0, which is correct as the replica has not yet received a full snapshot. The prepare is loaded into memory so the PDM has the correct state. On stream resumption the active now sends the following:
- SnapshotMarker 1-2 with flag Memory (and perhaps Checkpoint but that's orthogonal to this issue)
- DcpMutation for keyB with seqno2
At this point the PassiveStream would normally notify the PDM of a snapshot end as it has the final item and the snapshot contained a prepare. This is an in-memory variable though so post-restart the value is reset and a PassiveSteam processing these messages would not notify the PDM of the snapshot end as it has not seen a prepare. Disk snapshots always notify the PDM so they are not an issue here. As the PDM is not notified, the prepare at seqno 1 is not acked back to the Producer and this node cannot contribute towards it's commit. A subsequent prepare should fix this issue as the PassiveStream should notify the PDM of the snapshot in which it is contained.
Workaround
Perform another durable write to the affected vBucket and the original one will be "unstuck".
Solution
First thoughts here are that we should notify the PDM for all snapshots, but we didn't do that originally, perhaps to avoid some performance issue, so we could experiment with such a fix but may need to do something else.
Attachments
For Gerrit Dashboard: MB-51805 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
173645,2 | MB-51805: Do not notify pdm in ReceiveDuplicateDcpPrepareRemoveFromPDM | master | kv_engine | Status: ABANDONED | 0 | -1 |
173667,2 | MB-51805: Add highestTracked to DM interface | master | kv_engine | Status: ABANDONED | 0 | +1 |
173668,2 | MB-51805: Set PassiveStream::cur_snapshot_prepare on new stream | master | kv_engine | Status: ABANDONED | 0 | -1 |
173716,1 | MB-51805: Add persistedPrepareSeqno(maxPrepareSeqno) to makeVBucket() | master | kv_engine | Status: ABANDONED | 0 | -1 |
173717,1 | MB-51805: Pass persistedPrepareSeqno(maxPrepareSeqno) into VBucket | master | kv_engine | Status: ABANDONED | 0 | -1 |
173718,1 | MB-51805: Pass persistedPrepareSeqno(maxPrepareSeqno) to CkptMgr | master | kv_engine | Status: ABANDONED | -1 | -1 |
173911,4 | MB-51805: Do not notify pdm in ReceiveDuplicateDcpPrepareRemoveFromPDM | neo | kv_engine | Status: MERGED | +2 | +1 |
173912,4 | MB-51805: Add highestTracked to DM interface | neo | kv_engine | Status: MERGED | +2 | +1 |
173913,9 | MB-51805: Set PassiveStream::cur_snapshot_prepare on new stream | neo | kv_engine | Status: MERGED | +2 | +1 |
173914,5 | MB-51805: Add persistedPrepareSeqno(maxPrepareSeqno) to makeVBucket() | neo | kv_engine | Status: MERGED | +2 | +1 |
173915,5 | MB-51805: Pass persistedPrepareSeqno(maxPrepareSeqno) into VBucket | neo | kv_engine | Status: MERGED | +2 | +1 |
173916,6 | MB-51805: Pass persistedPrepareSeqno(maxPrepareSeqno) to CkptMgr | neo | kv_engine | Status: MERGED | +2 | +1 |
173917,6 | MB-51805: Pass persistedPrepareSeqno(maxPrepareSeqno) into Checkpoint | neo | kv_engine | Status: MERGED | +2 | +1 |
173918,6 | MB-51805, MB-51806: Inherit HPS from previous Checkpoint | neo | kv_engine | Status: MERGED | +2 | +1 |
174461,2 | MB-51805: Add Checkpoint::HPS value to dump() | neo | kv_engine | Status: MERGED | +2 | +1 |
174672,1 | Merge branch 'neo' | master | kv_engine | Status: ABANDONED | 0 | 0 |
174678,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175116,1 | MB-51805: Merge branch neo | master | kv_engine | Status: MERGED | +2 | +1 |
175807,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175816,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175862,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175867,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175933,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
175944,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |