Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
7.6.2
-
None
-
Untriaged
-
0
-
Unknown
Description
I see the following logs when deleting the buckets with enableCrossClusterVersioning set to true, with an active replication between the buckets:
goxdcr.log:2024-06-27T05:24:38.451-07:00 ERRO GOXDCR.GenericSupervisor: pipelineFullTopic=b49f7a266849de8ce239c1ffb72c00c9/test/test, PipelineSupervisor_b49f7a266849de8ce239c1ffb72c00c9/test/test Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline |
goxdcr.log:. errors_seen=map[TopoChangeDet:enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline] |
goxdcr.log:2024-06-27T05:24:38.451-07:00 INFO GOXDCR.ReplMgr: Supervisor PipelineSupervisor_b49f7a266849de8ce239c1ffb72c00c9/test/test of type *supervisor.GenericSupervisor reported errors map[TopoChangeDet:enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline] |
goxdcr.log:2024-06-27T05:24:38.452-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated with error(s) r.update_err_ch : TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline, current status=name={b49f7a266849de8ce239c1ffb72c00c9/test/test}, status={Pending}, errors={[{"time":"2024-06-27T05:24:38.452106373-07:00","errMsg":"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline"},{"time":"2024-06-27T05:24:32.115314022-07:00","errMsg":"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed."}]}, oldProgress={The runtime context has been started}, progress={Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline} |
goxdcr.log:2024-06-27T05:24:39.250-07:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=b49f7a266849de8ce239c1ffb72c00c9/test/test, b49f7a266849de8ce239c1ffb72c00c9/test/test-860534066 expvar=Stats for pipeline b49f7a266849de8ce239c1ffb72c00c9/test/test-860534066 {"Backfill Old Progress": "", "Backfill Progress": "", "Errors": "[{\"time\":\"2024-06-27T05:24:38.452106373-07:00\",\"errMsg\":\"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline\"},{\"time\":\"2024-06-27T05:24:32.115314022-07:00\",\"errMsg\":\"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed.\"}]", "OldProgress": "The runtime context has been started", "Overview": {"CurrentTime": 1719491079250521139, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 3072, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 0, "data_replicated_uncompress": 0, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 0, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 0, "docs_cloned": 0, "docs_failed_cr_source": 0, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 0, "docs_received_from_dcp": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 0, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 0, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 0, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 0, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 0, "set_failed_cr_source": 0, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 0, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 0, "wtavg_docs_latency": 0, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline", "Status": "Pending"} |
goxdcr.log:2024-06-27T05:25:04.022-07:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=b49f7a266849de8ce239c1ffb72c00c9/test/test, Stats for pipeline b49f7a266849de8ce239c1ffb72c00c9/test/test-860534066 {"Backfill Old Progress": "", "Backfill Progress": "", "Errors": "[{\"time\":\"2024-06-27T05:24:38.452106373-07:00\",\"errMsg\":\"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline\"},{\"time\":\"2024-06-27T05:24:32.115314022-07:00\",\"errMsg\":\"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed.\"}]", "OldProgress": "The runtime context has been started", "Overview": {"CurrentTime": 1719491079250521139, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 3072, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 0, "data_replicated_uncompress": 0, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 0, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 0, "docs_cloned": 0, "docs_failed_cr_source": 0, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 0, "docs_received_from_dcp": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 0, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 0, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 0, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 0, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 0, "set_failed_cr_source": 0, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 0, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 0, "wtavg_docs_latency": 0, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Received error report : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline", "Status": "Pending"} |
goxdcr.log:2024-06-27T05:25:04.034-07:00 INFO GOXDCR.PipelineMgr: Replication Status=name={b49f7a266849de8ce239c1ffb72c00c9/test/test}, status={Pending}, errors={[{"time":"2024-06-27T05:25:04.024118446-07:00","errMsg":"genericPipeline.startPartsWithTimeout:Execution timed out"},{"time":"2024-06-27T05:24:38.452106373-07:00","errMsg":"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline"},{"time":"2024-06-27T05:24:32.115314022-07:00","errMsg":"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed."}]}, oldProgress={Source nozzles have been closed}, progress={Pipeline has been stopped} |
goxdcr.log:2024-06-27T05:25:14.025-07:00 INFO GOXDCR.PipelineMgr: Replication status is updated now, current status=name={b49f7a266849de8ce239c1ffb72c00c9/test/test}, status={Pending}, errors={[{"time":"2024-06-27T05:25:04.024118446-07:00","errMsg":"genericPipeline.startPartsWithTimeout:Execution timed out"},{"time":"2024-06-27T05:24:38.452106373-07:00","errMsg":"TopoChangeDet : enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline"},{"time":"2024-06-27T05:24:32.115314022-07:00","errMsg":"dcp_b49f7a266849de8ce239c1ffb72c00c9/test/test_172.23.105.4:11210_0 : DCP upr feed has been closed."}]}, oldProgress={Source nozzles have been closed}, progress={Pipeline has been stopped} |
Changing enableCrossClusterVersioning from true isn't allowed, but the logs indicate otherwise, when the buckets are deleted before the replication.
Steps to reproduce:
- Create a bucket in source and target cluster
- Enable enableCrossClusterVersioning setting in both buckets
- Create a replication between the buckets
- Delete the buckets
- Error "errors_seen=map[TopoChangeDet:enableCrossClusterVersioning for source bucket has changed from true to false. Restarting pipeline]" seen in logs.