Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61359

[XDCR] - Backfill mutations stuck at non zero value

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.2.5
    • 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.4, 7.0.5, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.1.3, 7.2.1, 7.1.5, 7.2.4, 7.2.2, 7.1.6, 7.2.3, 7.2.5
    • XDCR
    • Untriaged
    • 0
    • No

    Description

      Backfill mutations are stuck at 31.8M mutations remaining, when target bucket is deleted and backfill mutations are high. The mutations remaining counter does not move, and is stuck at the same non zero value. There were some target topology changes while the replication was in progress as well.

      Steps to reproduce:

      1. Create 2 on-prem clusters of 2 nodes each
      2. Create a test bucket on each cluster
      3. Setup replication between the clusters with scopes and collections mapping set to a non default scope, post adding a remote cluster reference from source to target cluster
      4. Load documents in default scope and default collection, this will not get replicated for now since the scope and collection mapping is set for non default scope
      5. Now edit the replication and add default scope and collection in scopes and collections mapping
      6. Delete the target bucket post documents on source and target are synced, recreate bucket and replication with same name and settings
      7. Backfill mutations are stuck, number of docs on source and target are same.

      Snippet from the logs:

      # HELP xdcr_changes_left_total Given the vBuckets of this node, the number of sequence numbers that need to be processed (either replicated or handled) before catching up to the high sequence numbers for the vBuckets.
      # TYPE xdcr_changes_left_total gauge
      xdcr_changes_left_total {targetClusterUUID="da69a312ce9421936ede6d891380ab3b", sourceBucketName="test", targetBucketName="test", pipelineType="Backfill"} 11884587
      xdcr_changes_left_total {targetClusterUUID="da69a312ce9421936ede6d891380ab3b", sourceBucketName="test", targetBucketName="test", pipelineType="Main"} 0 

      Logs (All nodes on 7.2.5-7576)

      Source (172.23.105.4, 172.23.96.197): 
      https://cb-engineering.s3.amazonaws.com/MB-60859_src/collectinfo-2024-04-01T131500-ns_1%40172.23.105.4.zip
      https://cb-engineering.s3.amazonaws.com/MB-60859_src/collectinfo-2024-04-01T131500-ns_1%40172.23.96.197.zip

      Target (172.23.105.195, 172.23.96.183):

      https://cb-engineering.s3.amazonaws.com/MB-60859_dest/collectinfo-2024-04-01T131652-ns_1%40172.23.105.195.zip
      https://cb-engineering.s3.amazonaws.com/MB-60859_dest/collectinfo-2024-04-01T131652-ns_1%40172.23.96.183.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ayush.nayyar Ayush Nayyar
              ayush.nayyar Ayush Nayyar
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty