Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
Morpheus
-
None
-
Untriaged
-
1
-
No
Description
What's the issue?
PiTR history is not maintained during replication meaning that although the "active" data is safe, the PiTR related history is not.
This means that we can systematically wipe out PiTR history by failing over nodes (whilst correctly maintaining/protecting the active data).
This is best described with an example, please see the steps to reproduce.
Steps to reproduce?
1) Create a two node cluster
2) Create a PiTR aware bucket without replicas and 1s granularity *
3) Load 1K documents
3a) Repeat until we have a decent amount of history
4) Backup the data with PiTR enabled (in my case, I see ~5K items) **
5) Enable replicas
6) Rebalance
7) Gracefully fail over one node
8) Perform another full backup (in this case, I see ~3K)
At this point we've "observed" the issue, which is that replication is "lossy" with regards to PiTR history, for example, if we extend this scenario:
9) Add back node with "full" recovery
10) Gracefully failover the other node
11) Add back with "full" recovery
12) Perform a backup (in this case I see 1K items)
We can see that we're able to systematically remove/purge our PiTR history.
What's the expected outcome?
We should see that our PiTR history is maintained e.g. our backup should always backup the same amount of data.
* PiTR can be enabled at 1s granularity using the following command:
Enable PiTR |
curl -X POST http://localhost:9000/pools/default/buckets/default -u Administrator:asdasd -d pitrEnabled=true -d pitrMaxHistoryAge=86400 -d pitrGranularity=1
|
Please note that you will need to enable developer preview.
** PiTR backups can be take using the following command(s):
Configure PiTR Archive |
cbbackupmgr config -a $PATH_TO_ARCHIVE -r $REPO --point-in-time
|
Take a Full Backup |
cbbackupmgr backup -a $PATH_TO_ARCHIVE -r $REPO -c localhost:9000 -u Administrator -p asdasd --full-backup
|