Details
-
Bug
-
Resolution: Fixed
-
Critical
-
2.5.0
-
Security Level: Public
-
2.5.0-1059
-
Untriaged
-
Sprint 2 - March 11 - April 3
Description
Design limitation
-------------------------
There's currently a limitation with xdcr in cases of
1. destination failover
2. destination bucket flush
3. destination bucket delete & recreate
Items at destination may not match those at source. XDCR checkpointing is a mechanism that keeps track of destination vbucket state changes. However checkpointing(even _pre_replicate) for a vbucket does not happen unless the replicator for that vbucket is active although it's well past the time to checkpoint. A vbucket replicator becomes active only when the vbucket receives new mutations. As a result, unless all vbuckets have new mutations, all items will not be replicated. Hence the data loss.
Destination bucket recreate/flush
--------------------------------------------------
Refer MB-10457
Destination Failover
-------------------------------
I faced with the problem when worked with cbrecovery 'negative' scenarios
But tests can be simplified:
1)
cluster1: 10.3.4.144, 10.3.4.145,10.3.4.146
cluster2: 10.3.4.147, 10.3.4.148,10.3.4.149
2) load beer-sample on cluster1
3) create beer-sample on cluster2
4) create bidirectional XDCR replication cluster1<->cluster2 for bucket "beer-sample"
5) failover 2 nodes on cluster1 and rebalance
result:
data lost on cluster1 (about a third of the data to be lost)
but we still have bidirectional replication and I expect that data will be restored on cluster1 from cluster2
I believe that we should be able to automatically recover items from replication for such scenarios.
Attachments
Issue Links
- is duplicated by
-
MB-10988 [Tap CAPI XDCR] Flush/Delete and re-created destination buckets cause data loss at destination cluster
- Closed
- is triggering
-
MB-11076 XDCR checkpointing : would more _pre_replicate calls during replication help detect data losses at destination sooner?
- Closed
- relates to
-
MB-10457 XDCR: Some docs not replicated after deletion and recreation of destination bucket
- Closed
For Gerrit Dashboard: MB-10179 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
39655,2 | MB-10179: Enable Delete/flush destination buckets xdcr tests | master | testrunner | Status: MERGED | +2 | +1 |
39709,3 | Revert "MB-10179: Enable Delete/flush destination buckets xdcr tests". Disabling test as of now as MB-10179 is not fixed yet. | master | testrunner | Status: MERGED | +2 | +1 |
40360,3 | MB-10179: implemented CAPI _mass_vbopaque_check | master | ns_server | Status: MERGED | +2 | +1 |
40361,3 | MB-10179: [xdcr] publish more per-vbucket details with vb stats | master | ns_server | Status: MERGED | +2 | +1 |
40364,3 | MB-10179: check for vbopaque mismatch periodically | master | ns_server | Status: MERGED | +2 | +1 |
40580,2 | CBQE:2380 Add tests for new dev item MB-10179 to handle replication in idle-xdcr scenarios | master | testrunner | Status: MERGED | +2 | +1 |