Details
-
Task
-
Resolution: Fixed
-
Major
-
None
-
Security Level: Public
-
None
-
CBG Sprint 112
-
3
Description
`expectedSeqs` and `processedSeqs` in the ISGR checkpointer maintain full lists of sequence IDs that have yet to be checkpointed but have been replicated/processed.
In normal circumstances, these lists are at most expected to be 2x the number of in-flight revisions. In abnormal cases, these lists can quite quickly grow as the replication fails to checkpoint any further yet continues to progress. This has adverse effects on performance and eventually makes the replication grind to a halt with the size of entries in these sets.
In many cases, we're able to safely compact away most of these sequences without impacting checkpointing by retaining the last processed sequence, and dropping all of those preceeding, until we hit a sequence that is expected but not yet processed.
As soon as the missing expected sequence arrives, we're able to checkpoint the retained sequence without worrying about any intermediate sequences.
Attachments
Issue Links
- backports to
-
CBG-2704 [2.8.4 Backport] ISGR Sequence checkpointing maintains unnessesary entries
-
- Closed
-
-
CBG-2705 [3.0.5] ISGR Sequence checkpointing maintains unnessesary entries
-
- Closed
-
- is cloned by
-
CBG-2649 [2.8.3.4 Backport] ISGR Sequence checkpointing maintains unnessesary entries
-
- Closed
-
- relates to
-
CBG-2556 Inefficient sequence parsing during ISGR checkpointing
-
- Closed
-