Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51526

[CBM] [PiTR] Same snapshot being streamed repeatedly

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Major
    • None
    • 7.1.0
    • tools
    • None
    • Untriaged
    • 1
    • No

    Description

      What's the issue?
      Whilst playing around with PiTR, I noticed that CBM was streaming the same snapshot continuously in a loop eventually breaking out after >6K attempts.

      Creating DCP Streams

      $ rg --fixed-strings 'Creating DCP stream | {"uuid":250230374546733,"start_seqno":375000,"end_seqno":377760,"snap_start":0,"snap_end":375000,"retries":0}' ~/Projects/couchbase-archive/logs/backup-0.log -c
      6717
      

      Explanation
      When streaming PiTR snapshots the simplified process is as follows:

      1. Get some stats from the server, including the high seqno for the vbucket
      2. Ask the server for a stream from disk with a start seqno of the last seen seqno and end seqno of the high seqno of the vbucket
      3. Persist the resulting snapshot
      4. If the snapshot ends at or beyond the high seqno then stop, otherwise goto 2.

      Consider what happens if disk persistence is a little bit behind in memory. In that case we might ask for a start-end seqno range that hasn't been persisted to disk yet. In that case the server will just not give us a snapshot and, as there is no new information, the check at 4 will goto 2 and we'll retry until some more data is flushed.

      Solution
      We believe the best way to solve this may be with MB-46854 (stream all snapshot). Then once the server has read everything it has on disk it can close the stream and we'll terminate. Alternatively we may be able to work around this in cbbackupmgr

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty