Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62517

[XDCR] - logging source ECCV changed from true to false, which isn't possible

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • 7.6.4
    • 7.6.2
    • XDCR
    • None

    Description

      I see the following logs when deleting the buckets with enableCrossClusterVersioning set to true, with an active replication between the buckets:

      goxdcr.log:2024-06-27T05:24:38.451-07:00 ERRO GOXDCR.GenericSupervisor: pipelineFullTopic=b49f7a266849de8ce239c1ffb72c00c9/test/test, PipelineSupervisor_b49f7a266849de8ce239c1ffb72c00c9/test/test Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline
      goxdcr.log:. errors_seen=map[TopoChangeDet:enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline]
      goxdcr.log:2024-06-27T05:24:38.451-07:00 INFO GOXDCR.ReplMgr: Supervisor PipelineSupervisor_b49f7a266849de8ce239c1ffb72c00c9/test/test of type *supervisor.GenericSupervisor reported errors map[TopoChangeDet:enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline]
      goxdcr.log:2024-06-27T05:24:38.452-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) r.update_err_ch : TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline, current status=name={b49f7a266849de8ce239c1ffb72c00c9/test/test}, status={Pending}, errors={[{"time":"2024-06-27T05:24:38.452106373-07:00","errMsg":"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline"},{"time":"2024-06-27T05:24:32.115314022-07:00","errMsg":"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed."}]}, oldProgress={The runtime context has been started}, progress={Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline}
      goxdcr.log:2024-06-27T05:24:39.250-07:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=b49f7a266849de8ce239c1ffb72c00c9/test/test, b49f7a266849de8ce239c1ffb72c00c9/test/test-860534066 expvar=Stats for pipeline b49f7a266849de8ce239c1ffb72c00c9/test/test-860534066 {"Backfill Old Progress": "", "Backfill Progress": "", "Errors": "[{\"time\":\"2024-06-27T05:24:38.452106373-07:00\",\"errMsg\":\"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline\"},{\"time\":\"2024-06-27T05:24:32.115314022-07:00\",\"errMsg\":\"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed.\"}]", "OldProgress": "The runtime context has been started", "Overview": {"CurrentTime": 1719491079250521139, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 3072, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 0, "data_replicated_uncompress": 0, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 0, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 0, "docs_cloned": 0, "docs_failed_cr_source": 0, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 0, "docs_received_from_dcp": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 0, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 0, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 0, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 0, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 0, "set_failed_cr_source": 0, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 0, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 0, "wtavg_docs_latency": 0, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline", "Status": "Pending"}
      goxdcr.log:2024-06-27T05:25:04.022-07:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=b49f7a266849de8ce239c1ffb72c00c9/test/test, Stats for pipeline b49f7a266849de8ce239c1ffb72c00c9/test/test-860534066 {"Backfill Old Progress": "", "Backfill Progress": "", "Errors": "[{\"time\":\"2024-06-27T05:24:38.452106373-07:00\",\"errMsg\":\"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline\"},{\"time\":\"2024-06-27T05:24:32.115314022-07:00\",\"errMsg\":\"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed.\"}]", "OldProgress": "The runtime context has been started", "Overview": {"CurrentTime": 1719491079250521139, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 3072, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 0, "data_replicated_uncompress": 0, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 0, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 0, "docs_cloned": 0, "docs_failed_cr_source": 0, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 0, "docs_received_from_dcp": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 0, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 0, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 0, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 0, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 0, "set_failed_cr_source": 0, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 0, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 0, "wtavg_docs_latency": 0, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline", "Status": "Pending"}
      goxdcr.log:2024-06-27T05:25:04.034-07:00 INFO GOXDCR.PipelineMgr: Replication Status=name={b49f7a266849de8ce239c1ffb72c00c9/test/test}, status={Pending}, errors={[{"time":"2024-06-27T05:25:04.024118446-07:00","errMsg":"genericPipeline.startPartsWithTimeout:Execution timed out"},{"time":"2024-06-27T05:24:38.452106373-07:00","errMsg":"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline"},{"time":"2024-06-27T05:24:32.115314022-07:00","errMsg":"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed."}]}, oldProgress={Source nozzles have been closed}, progress={Pipeline has been stopped}
      goxdcr.log:2024-06-27T05:25:14.025-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated now, current status=name={b49f7a266849de8ce239c1ffb72c00c9/test/test}, status={Pending}, errors={[{"time":"2024-06-27T05:25:04.024118446-07:00","errMsg":"genericPipeline.startPartsWithTimeout:Execution timed out"},{"time":"2024-06-27T05:24:38.452106373-07:00","errMsg":"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline"},{"time":"2024-06-27T05:24:32.115314022-07:00","errMsg":"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed."}]}, oldProgress={Source nozzles have been closed}, progress={Pipeline has been stopped}

      Changing enableCrossClusterVersioning from true isn't allowed, but the logs indicate otherwise, when the buckets are deleted before the replication. 

      Steps to reproduce:

      1. Create a bucket in source and target cluster
      2. Enable enableCrossClusterVersioning setting in both buckets
      3. Create a replication between the buckets
      4. Delete the buckets
      5. Error "errors_seen=map[TopoChangeDet:enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline]" seen in logs.

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ayush.nayyar Ayush Nayyar
            ayush.nayyar Ayush Nayyar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty