Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-2645

ISGR Sequence checkpointing maintains unnessesary entries

    XMLWordPrintable

Details

    • Task
    • Resolution: Fixed
    • Major
    • 3.1.0
    • None
    • SyncGateway
    • Security Level: Public
    • None
    • CBG Sprint 112
    • 3

    Description

      `expectedSeqs` and `processedSeqs` in the ISGR checkpointer maintain full lists of sequence IDs that have yet to be checkpointed but have been replicated/processed.

      In normal circumstances, these lists are at most expected to be 2x the number of in-flight revisions. In abnormal cases, these lists can quite quickly grow as the replication fails to checkpoint any further yet continues to progress. This has adverse effects on performance and eventually makes the replication grind to a halt with the size of entries in these sets.

      In many cases, we're able to safely compact away most of these sequences without impacting checkpointing by retaining the last processed sequence, and dropping all of those preceeding, until we hit a sequence that is expected but not yet processed.

      As soon as the missing expected sequence arrives, we're able to checkpoint the retained sequence without worrying about any intermediate sequences.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ben.brooks Ben Brooks
              ben.brooks Ben Brooks
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty