Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.4, 7.0.5, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.1.3, 7.2.1, 7.1.5, 7.2.4, 7.2.2, 7.1.6, 7.2.3, 7.2.5
-
Untriaged
-
0
-
No
Description
Backfill mutations are stuck at 31.8M mutations remaining, when target bucket is deleted and backfill mutations are high. The mutations remaining counter does not move, and is stuck at the same non zero value. There were some target topology changes while the replication was in progress as well.
Steps to reproduce:
- Create 2 on-prem clusters of 2 nodes each
- Create a test bucket on each cluster
- Setup replication between the clusters with scopes and collections mapping set to a non default scope, post adding a remote cluster reference from source to target cluster
- Load documents in default scope and default collection, this will not get replicated for now since the scope and collection mapping is set for non default scope
- Now edit the replication and add default scope and collection in scopes and collections mapping
- Delete the target bucket post documents on source and target are synced, recreate bucket and replication with same name and settings
- Backfill mutations are stuck, number of docs on source and target are same.
Snippet from the logs:
# HELP xdcr_changes_left_total Given the vBuckets of this node, the number of sequence numbers that need to be processed (either replicated or handled) before catching up to the high sequence numbers for the vBuckets. |
# TYPE xdcr_changes_left_total gauge
|
xdcr_changes_left_total {targetClusterUUID="da69a312ce9421936ede6d891380ab3b", sourceBucketName="test", targetBucketName="test", pipelineType="Backfill"} 11884587 |
xdcr_changes_left_total {targetClusterUUID="da69a312ce9421936ede6d891380ab3b", sourceBucketName="test", targetBucketName="test", pipelineType="Main"} 0 |
Logs (All nodes on 7.2.5-7576)
Source (172.23.105.4, 172.23.96.197):
https://cb-engineering.s3.amazonaws.com/MB-60859_src/collectinfo-2024-04-01T131500-ns_1%40172.23.105.4.zip
https://cb-engineering.s3.amazonaws.com/MB-60859_src/collectinfo-2024-04-01T131500-ns_1%40172.23.96.197.zip
Target (172.23.105.195, 172.23.96.183):
https://cb-engineering.s3.amazonaws.com/MB-60859_dest/collectinfo-2024-04-01T131652-ns_1%40172.23.105.195.zip
https://cb-engineering.s3.amazonaws.com/MB-60859_dest/collectinfo-2024-04-01T131652-ns_1%40172.23.96.183.zip
Attachments
Issue Links
- is a backport of
-
MB-61378 XDCR - Backfill pipeline stats are not reset if replication was deleted when backfill is in progress (negative changes_left)
- Closed