Major Description
Problem
When --force-update is used, cbbackupmgr sets the force flags on the META operations. This means conflict resolution is skipped and all data is accepted. This does not play well with XDCR
Suggestion
When --force-update is used the the REGENERATE_CAS on the META operations should be set.
Steps to reproduce
1. Setup two one-node clusters (Cluster A should have the backup service enabled)
2. Setup XDCR between the two clusters - single directional (Cluster A -> Cluster B)
3. Create a doc on cluster A and ensure it appears on cluster B
4. Take a backup off cluster A
5. Delete the document, again ensure it's deleted on cluster B
6. Do a restore using --force-update against cluster A
7. Cluster A will have the document Cluster B will not. The two clusters are now out of sync, this is the problem.
Notes
The use case should be better understood. The general case is when documents have accidentally been deleted and the user wants a restore that just replaces the deleted documents. Maybe there is an argument of having that mode natively rather than using --force-update with the --key-filter. That will need protocol changes as XATTR support on normal commands is limited.
Fixed
This issue has been fixed in an indirect way so it's worth explaining how to to setup XDCR with this fix to work properly for this use case. Even with the fix, following the steps to reproduce above will result in the same outcome as before. This is due to the fact that the default conflict resolution mode on created buckets is "Sequence number", which prioritises Sequence numbers of documents instead of CAS values. In order to make XDCR work as expected for this use case the buckets in the XDCR connection should be set up with the alternative conflict resolution mode "Timestamp", which uses CAS values before all else.
Steps to confirm that the fix works:
1. Setup two one-node clusters (Cluster A should have the backup service enabled)
2. (!!!) When creating buckets for the XDCR connection on both clusters, go into Advanced bucket settings and in Conflict resolution select "Timestamp"
3. Setup XDCR between the two clusters - single directional (Cluster A -> Cluster B)
4. Create a doc on cluster A and ensure it appears on cluster B
5. Take a backup off cluster A
6. Delete the document, again ensure it's deleted on cluster B
7. Do a restore using --force-update against cluster A
8. Cluster A will have the document and so will Cluster B.