Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51033

[PiTR] Initial DCP replica build doesn't maintain PiTR history

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Morpheus
    • Morpheus
    • couchbase-bucket
    • None
    • Untriaged
    • 1
    • No

    Description

      What's the issue?
      PiTR history is not maintained during replication meaning that although the "active" data is safe, the PiTR related history is not.

      This means that we can systematically wipe out PiTR history by failing over nodes (whilst correctly maintaining/protecting the active data).

      This is best described with an example, please see the steps to reproduce.

      Steps to reproduce?
      1) Create a two node cluster
      2) Create a PiTR aware bucket without replicas and 1s granularity *
      3) Load 1K documents
      3a) Repeat until we have a decent amount of history
      4) Backup the data with PiTR enabled (in my case, I see ~5K items) **
      5) Enable replicas
      6) Rebalance
      7) Gracefully fail over one node
      8) Perform another full backup (in this case, I see ~3K)

      At this point we've "observed" the issue, which is that replication is "lossy" with regards to PiTR history, for example, if we extend this scenario:

      9) Add back node with "full" recovery
      10) Gracefully failover the other node
      11) Add back with "full" recovery
      12) Perform a backup (in this case I see 1K items)

      We can see that we're able to systematically remove/purge our PiTR history.

      What's the expected outcome?
      We should see that our PiTR history is maintained e.g. our backup should always backup the same amount of data.

      * PiTR can be enabled at 1s granularity using the following command:

      Enable PiTR

      curl -X POST http://localhost:9000/pools/default/buckets/default -u Administrator:asdasd -d pitrEnabled=true -d pitrMaxHistoryAge=86400 -d pitrGranularity=1
      

      Please note that you will need to enable developer preview.

      ** PiTR backups can be take using the following command(s):

      Configure PiTR Archive

      cbbackupmgr config -a $PATH_TO_ARCHIVE -r $REPO --point-in-time
      

      Take a Full Backup

      cbbackupmgr backup -a $PATH_TO_ARCHIVE -r $REPO -c localhost:9000 -u Administrator -p asdasd --full-backup
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            owend Daniel Owen
            james.lee James Lee
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty