2024-05-31T11:19:01.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:01.694Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:02.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:02.697Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:03.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:03.645Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:04.375Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:19:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:19:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:19:04.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:05.302Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:05.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:06.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:06.373Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:06.373Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:06.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:07.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:07.645Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:08.244Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:08.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:09.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:09.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:10.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:10.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:11.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:11.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:12.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:12.503Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:13.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:13.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:14.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:14.398Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:19:14.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:19:14.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:19:14.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:19:14.483Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:19:14.521Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:19:14.557Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:19:14.557Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:19:14.557Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:19:14.568Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:19:14.573Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:19:14.590Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:15.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:15.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:16.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:16.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:16.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:16.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:17.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:17.588Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:18.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:18.599Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:19.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:19.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:20.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:20.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:21.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:21.539Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:22.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:22.649Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:23.309Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154362649676571, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:19:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:19:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:19:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:19:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:19:23.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=390.162µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:19:23.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:24.196Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:24.407Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:19:24.407Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:19:24.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:19:24.531Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:25.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:25.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:26.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:26.375Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:26.376Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:26.699Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:27.327Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:27.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:28.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:28.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:29.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:29.417Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:19:29.490Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:19:29.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:19:29.517Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:19:29.517Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:19:29.517Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:19:29.523Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:19:29.543Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:19:29.618Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:30.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:30.445Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:31.149Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 224, "num_failedckpts": 0, "time_committing": {"count": 224, "max": 6923, "mean": 76.69642857142857, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154370252815064, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 224, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:19:31.150Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:19:31.150Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:19:31.151Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:19:31.152Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:19:31.165Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=13.06427ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:19:31.324Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:31.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:32.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:32.593Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:33.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:33.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:34.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:19:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:19:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:19:34.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:35.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:35.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:36.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:36.417Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:36.417Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:36.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:37.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:37.802Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:38.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:38.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:39.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:39.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:40.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:40.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:41.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:41.624Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:42.440Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:42.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:43.157Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:43.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:44.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:44.399Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:19:44.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:19:44.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:19:44.400Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:19:44.479Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:19:44.507Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:19:44.563Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:19:44.615Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:45.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:45.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:46.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:46.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:46.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:46.459Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:47.407Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:47.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:48.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:48.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:49.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:49.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:50.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:50.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:51.329Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:51.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:52.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:52.676Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:53.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154392677078578, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:19:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:19:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:19:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:19:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:19:53.388Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=726.444µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:19:53.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:54.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:54.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:19:54.400Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:19:54.401Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:19:54.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:55.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:55.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:56.229Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:56.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:56.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:19:56.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:57.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:57.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:58.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:58.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:19:59.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:19:59.401Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:19:59.451Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:19:59.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:19:59.554Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:20:00.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:00.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:01.147Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 224, "num_failedckpts": 0, "time_committing": {"count": 224, "max": 6923, "mean": 76.69642857142857, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154400276035959, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 224, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:20:01.151Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:01.151Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:01.151Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:20:01.151Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:20:01.152Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=517.59µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:20:01.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:01.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:02.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:02.622Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:03.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:03.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:04.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:04.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:20:04.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:20:04.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:20:04.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:05.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:05.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:06.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:06.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:06.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:06.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:07.381Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:07.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:08.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:08.594Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:10.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:10.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:11.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:11.379Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:12.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:12.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:13.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:13.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:14.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:14.379Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:14.397Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":131201240,"TotalAlloc":90938228746096,"Sys":327718216,"Lookups":0,"Mallocs":968723227951,"Frees":968721919965,"HeapAlloc":131201240,"HeapSys":308477952,"HeapIdle":154771456,"HeapInuse":153706496,"HeapReleased":92897280,"HeapObjects":1307986,"StackInuse":1900544,"StackSys":1900544,"MSpanInuse":2460480,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191234,"GCSys":7331136,"OtherSys":1611990,"NextGC":187013472,"LastGC":1717154414346197264,"PauseTotalNs":3771362871295,"PauseNs":[9742046,8046580,13747054,26568072,8650052,11091809,126133,11900550,10379716,12290698,29006912,144944,12953298,22954982,6665595,12943243,12000548,12251342,4039528,4070840,144288,12133568,11079505,181473,12980288,21322289,28823782,5345765,6459563,15871659,2488471,149475,124822,10702745,28046400,2679139,12054928,10451642,16124881,4048129,95297,16041749,8140536,12148680,22698848,9024744,18301974,8164203,12017281,12482228,23875462,26327926,18316984,29675027,8826119,24075096,19000721,15861815,24940895,19970868,12087056,9522155,18523816,43349060,19100209,15481309,20743602,7582182,93934,204094,4316495,20133777,138000,7659445,8160427,5676382,19952788,11284578,425706,9870196,19605037,8073896,96734,15137970,4468719,8636321,156794,5547737,10253363,16390559,10691253,25577379,7697399,7648038,19941785,3120711,2706255,15725529,19915330,6647573,12421563,139640,12197951,7454682,12726880,132329,15398728,15168458,16270255,8448900,14006948,2749791,4090316,15190754,10291239,4153511,17618375,23659574,19232305,21092499,32724228,19720440,26127913,8864803,17307633,22152471,8019009,15356631,157451,12218815,12207754,15430629,19992424,90151,3370832,9816952,16285495,8114517,17306979,11753522,9128983,13592617,218774,2224586,3372067,345189,164606,12381082,11833876,139003,4152346,22379197,16197154,169599,9010093,153008,154472,145398,7665643,162786,7962503,8318573,136956,130411,38457194,12156095,8129247,18355909,7247409,14117452,12048961,23867880,9131051,3639287,6672533,165370,142809,134923,138570,132783,133381,118991,143367,140743,132644,127662,138003,122353,133567,115396,133911,302495,132754,145654,137404,136651,133571,127811,135888,366633,135318,131084,120708,126279,114256,153253,136041,126796,120843,135328,143905,122490,144447,142617,305770,133539,144834,4232641,8469556,12506914,11924130,3581540,13696117,3820655,15957896,22760984,12090550,14253196,7990156,12123782,3170620,22031351,20028167,22573067,25286636,8127923,10533965,127947,8011281,16106980,4724357,21751776,10414645,11434147,156892,8028542,139956,140201,11419254,1766445,9869291,16945156,24563283,24792833,12210278,20818911],"PauseEnd":[1717154305828879516,1717154306386390340,1717154306903208896,1717154307592622888,1717154308163020558,1717154308691016303,1717154309187194237,1717154309743867026,1717154310337095280,1717154310889956381,1717154311462979799,1717154311960244342,1717154312682809404,1717154313194031255,1717154313739173872,1717154314271890427,1717154314816510560,1717154315401467932,1717154315922042838,1717154316486934514,1717154316956822036,1717154317721149711,1717154318286833652,1717154318791512972,1717154319312127289,1717154319820788201,1717154320441776665,1717154320950714629,1717154321579555217,1717154322132418147,1717154322763016960,1717154323358640598,1717154323891171220,1717154324528082051,1717154325040719042,1717154325649881892,1717154326174707448,1717154326723789272,1717154327324713620,1717154327981623324,1717154328563136205,1717154329075622474,1717154329644577662,1717154330205607268,1717154330842678326,1717154331415854189,1717154331900320882,1717154332597506367,1717154333126464261,1717154333695733962,1717154334212007444,1717154334716794834,1717154335285771572,1717154335870780755,1717154336431630358,1717154336924010671,1717154337641415976,1717154338194177821,1717154338747279753,1717154339336093799,1717154339917038447,1717154340498390861,1717154341071387400,1717154342138378940,1717154343406740119,1717154344339019785,1717154345126939432,1717154345822464763,1717154346393055894,1717154346869230550,1717154347530364073,1717154348113623397,1717154348656913118,1717154349196439800,1717154349763933574,1717154350328006152,1717154350894942587,1717154351540977817,1717154352087431608,1717154352770825691,1717154353382148765,1717154353913581244,1717154354505272881,1717154355032016716,1717154355615225517,1717154356174486094,1717154356737726763,1717154357386221292,1717154358037606776,1717154358640919231,1717154359126384742,1717154359717079689,1717154360247269573,1717154360762393951,1717154361355697950,1717154361892220442,1717154362611496894,1717154363222178829,1717154363821536421,1717154364373506954,1717154364870023760,1717154365414124940,1717154365927124701,1717154366587394247,1717154367254651671,1717154367885531698,1717154368452155735,1717154368947042754,1717154369569681098,1717154370064911975,1717154370662933524,1717154371217584732,1717154371824163205,1717154372539997178,1717154373147892255,1717154373750614886,1717154374289662477,1717154374828348606,1717154375407344316,1717154375934097802,1717154376545682045,1717154377330977035,1717154378078864874,1717154378681163304,1717154379198124417,1717154379789004594,1717154380411799611,1717154380946603601,1717154381585995652,1717154382240687527,1717154382905302389,1717154383480726495,1717154383975338622,1717154384562159724,1717154385073477306,1717154385628720924,1717154386155471962,1717154386703164164,1717154387278631366,1717154387987522931,1717154388603258384,1717154389097137671,1717154389652683925,1717154390125334616,1717154390664257046,1717154391221857503,1717154391781080860,1717154392508623873,1717154393172671550,1717154393755797871,1717154394347113605,1717154394914363380,1717154395529101336,1717154396018307798,1717154396636670466,1717154397175000348,1717154397911227616,1717154398475451196,1717154399003573208,1717154399626443647,1717154400173091467,1717154400735029940,1717154401304230827,1717154401873558272,1717154402555113769,1717154403188959870,1717154403806737664,1717154404405516798,1717154404963856202,1717154405596933413,1717154406107095982,1717154406706393474,1717154407301388782,1717154407962512918,1717154408567512203,1717154408990428458,1717154409119474701,1717154409257341240,1717154409389524807,1717154409523915964,1717154409656683829,1717154409789930649,1717154409922686562,1717154410056655123,1717154410182974495,1717154410308866792,1717154410435710023,1717154410563854154,1717154410690666547,1717154410817244541,1717154410944358895,1717154411089908335,1717154411223321902,1717154411358468900,1717154411492867847,1717154411625782376,1717154411756841910,1717154411890750324,1717154412021812258,1717154412153470708,1717154412297573764,1717154412423457859,1717154412549114367,1717154412676083637,1717154412799582680,1717154412926912395,1717154413065249359,1717154413181216126,1717154413306110097,1717154413426507435,1717154413546924599,1717154413670299142,1717154413804049765,1717154413937104624,1717154414084785219,1717154414219971553,1717154414346197264,1717154283756387134,1717154284379704009,1717154284939280010,1717154285517624062,1717154285968238632,1717154286538739828,1717154287120539802,1717154287812511159,1717154288351589430,1717154288854482160,1717154289399927766,1717154289962281023,1717154290573333592,1717154291066980613,1717154291670513652,1717154292177755327,1717154292837262323,1717154293368439330,1717154293875687475,1717154294451574271,1717154295014541049,1717154295641352582,1717154296192743492,1717154296780473679,1717154297532626644,1717154298024386672,1717154298611333737,1717154299099294856,1717154299639040535,1717154300209348926,1717154300771486566,1717154301357186108,1717154301883576184,1717154302580093974,1717154303098545373,1717154303688549223,1717154304206004772,1717154304759044868,1717154305344184883],"NumGC":1076185,"NumForcedGC":0,"GCCPUFraction":0.03631596186124228,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":436960225,"Frees":436953589},{"Size":16,"Mallocs":6761950128,"Frees":6761587758},{"Size":24,"Mallocs":19126808959,"Frees":19126658754},{"Size":32,"Mallocs":10352409478,"Frees":10352321971},{"Size":48,"Mallocs":1987652767,"Frees":1987604700},{"Size":64,"Mallocs":240386610,"Frees":240354450},{"Size":80,"Mallocs":238664940,"Frees":238649706},{"Size":96,"Mallocs":914185185165,"Frees":914184681186},{"Size":112,"Mallocs":383013112,"Frees":383009145},{"Size":128,"Mallocs":9665061225,"Frees":9665046567},{"Size":144,"Mallocs":190011168,"Frees":190006773},{"Size":160,"Mallocs":71529675,"Frees":71527686},{"Size":176,"Mallocs":5492939,"Frees":5492139},{"Size":192,"Mallocs":32335413,"Frees":32334963},{"Size":208,"Mallocs":68188241,"Frees":68184257},{"Size":224,"Mallocs":3198052,"Frees":3197526},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52350083,"Frees":52322536},{"Size":288,"Mallocs":98431642,"Frees":98409550},{"Size":320,"Mallocs":49943307,"Frees":49939828},{"Size":352,"Mallocs":15118224,"Frees":15117657},{"Size":384,"Mallocs":30400038,"Frees":30399859},{"Size":416,"Mallocs":5921082,"Frees":5918586},{"Size":448,"Mallocs":5902057,"Frees":5897750},{"Size":480,"Mallocs":642393,"Frees":642041},{"Size":512,"Mallocs":39553353,"Frees":39553143},{"Size":576,"Mallocs":21062806,"Frees":21060375},{"Size":640,"Mallocs":1231695,"Frees":1231450},{"Size":704,"Mallocs":18195274,"Frees":18194560},{"Size":768,"Mallocs":6969955,"Frees":6969624},{"Size":896,"Mallocs":7310659,"Frees":7310253},{"Size":1024,"Mallocs":52492524,"Frees":52492345},{"Size":1152,"Mallocs":24928965,"Frees":24928895},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16232771,"Frees":16232742},{"Size":1536,"Mallocs":1797847,"Frees":1797843},{"Size":1792,"Mallocs":12359051,"Frees":12356960},{"Size":2048,"Mallocs":709065,"Frees":708969},{"Size":2304,"Mallocs":22902088,"Frees":22901568},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2035492,"Frees":2035475},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8451044,"Frees":8451029},{"Size":4096,"Mallocs":3533881,"Frees":3533793},{"Size":4864,"Mallocs":8014666,"Frees":8014632},{"Size":5376,"Mallocs":1747682,"Frees":1746138},{"Size":6144,"Mallocs":1307329,"Frees":1307319},{"Size":6528,"Mallocs":6054185,"Frees":6054176},{"Size":6784,"Mallocs":1755846,"Frees":1755773},{"Size":6912,"Mallocs":249567,"Frees":249567},{"Size":8192,"Mallocs":225827,"Frees":225725},{"Size":9472,"Mallocs":1836106,"Frees":1836086},{"Size":9728,"Mallocs":106453,"Frees":106453},{"Size":10240,"Mallocs":1187400,"Frees":1187399},{"Size":10880,"Mallocs":33504,"Frees":33504},{"Size":12288,"Mallocs":272685,"Frees":272681},{"Size":13568,"Mallocs":7809720,"Frees":7809570},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1387326,"Frees":1387296},{"Size":18432,"Mallocs":1188755,"Frees":1188751}]} 2024-05-31T11:20:14.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:20:14.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:20:14.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:20:14.399Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:20:14.415Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:20:14.431Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:20:14.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:20:14.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:20:14.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:20:14.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:20:14.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:20:15.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:15.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:16.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:16.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:16.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:16.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:17.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:17.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:18.099Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:18.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:19.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:19.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:20.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:20.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:21.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:21.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:22.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:22.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:23.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:23.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154423371872709, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:20:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:23.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:20:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:20:23.390Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=481.163µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:20:24.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:24.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:24.404Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:20:24.404Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:20:24.404Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:20:25.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:25.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:26.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:26.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:26.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:26.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:27.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:27.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:28.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:28.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:29.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:29.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:20:29.414Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:20:29.431Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:20:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:20:29.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:20:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:20:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:20:29.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:20:30.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:30.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:31.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 224, "num_failedckpts": 0, "time_committing": {"count": 224, "max": 6923, "mean": 76.69642857142857, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154431087736049, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 224, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:20:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:20:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:20:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=479.231µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:20:31.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:32.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:32.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:33.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:33.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:34.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:34.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:20:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:20:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:20:35.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:35.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:36.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:36.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:36.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:37.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:37.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:38.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:38.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:39.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:39.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:40.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:40.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:41.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:41.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:42.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:42.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:43.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:43.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:44.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:44.383Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:44.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:20:44.404Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:20:44.404Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:20:44.404Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:20:44.421Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:20:44.436Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:20:44.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:20:45.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:45.390Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:46.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:46.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:46.329Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:46.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:47.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:47.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:48.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:48.396Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:49.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:50.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:50.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:51.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:51.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:52.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:52.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:53.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:53.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:53.393Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154453362018855, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:20:53.393Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:53.393Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:20:53.393Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:20:53.393Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:20:53.394Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=381.202µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:20:54.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:54.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:54.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:20:54.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:20:54.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:20:55.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:55.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:56.089Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:56.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:56.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:20:56.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:57.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:57.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:58.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:58.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:59.089Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:20:59.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:20:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:20:59.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:20:59.427Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:20:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:20:59.497Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:20:59.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:20:59.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:20:59.499Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:21:00.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:00.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:01.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 224, "num_failedckpts": 0, "time_committing": {"count": 224, "max": 6923, "mean": 76.69642857142857, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154461071316878, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 224, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:21:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:21:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:21:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=516.273µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:21:01.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:02.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:02.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:03.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:03.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:04.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:04.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:21:04.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:21:04.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:21:05.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:05.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:06.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:06.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:06.335Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:06.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:07.102Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:07.383Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:08.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:08.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:09.129Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:09.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:10.119Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:10.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:11.116Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:12.153Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:12.440Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:13.414Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:14.319Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:14.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:21:14.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:21:14.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:21:14.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:21:14.422Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:21:14.474Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:21:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:21:14.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:15.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:15.533Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:16.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:16.384Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:16.384Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:16.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:17.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:17.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:18.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:18.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:19.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:19.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:20.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:20.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:21.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:21.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:22.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:22.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:23.313Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154482587579781, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:21:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:21:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:21:23.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=366.959µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:21:23.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:24.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:21:24.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:21:24.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:21:24.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:25.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:25.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:26.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:26.382Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:26.383Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:26.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:27.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:27.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:28.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:28.454Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:29.254Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:21:29.438Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:21:29.472Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:21:29.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:21:29.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:21:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:21:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:21:29.501Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:21:30.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:30.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:31.138Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 224, "num_failedckpts": 0, "time_committing": {"count": 224, "max": 6923, "mean": 76.69642857142857, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154490292199558, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 224, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:21:31.166Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:31.166Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:31.167Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:21:31.167Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:21:31.167Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=480.639µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:21:31.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:31.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:32.237Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:32.624Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:33.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:33.528Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:33.819Z ERRO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Pipeline has not been started (or has been disabled) - checkpointing is skipped 2024-05-31T11:21:34.319Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:34.400Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:21:34.400Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:21:34.400Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:21:34.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:35.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:35.488Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Done checkpointing with vb list [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511] totalSize=0 2024-05-31T11:21:35.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:36.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:36.305Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Done checkpointing for replication 2024-05-31T11:21:36.305Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt.disableCkptAndWait took 8m41.562706369s 2024-05-31T11:21:36.309Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:36.310Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:36.310Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:21:36.311Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:36.311Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:36.383Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:36.383Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:36.433Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:36.446Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:36.446Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:21:36.675Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:37.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:37.585Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:37.626Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:37.950Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:38.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:38.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:39.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:39.520Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:40.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:40.627Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:41.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:41.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:42.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:42.608Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:43.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:43.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:44.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:44.401Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:21:44.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:21:44.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:21:44.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:21:44.504Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:21:44.517Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:21:44.517Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:21:44.517Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:21:44.519Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:21:44.549Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:21:44.595Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:21:44.639Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:45.335Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:45.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:46.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:46.341Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:46.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:46.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:47.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:47.630Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:48.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:48.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:49.394Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:49.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:49.600Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:50.229Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:50.346Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:21:50.346Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:21:50.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:50.612Z WARN GOXDCR.Utils: ckmgr.mergeAndPersistBrokenMappingDocsAndCkpts took 13.042125884s 2024-05-31T11:21:51.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:51.495Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt took 8m56.752503404s 2024-05-31T11:21:51.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:52.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:52.650Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:53.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154512650879150, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:21:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:21:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:21:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:21:53.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=383.471µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:21:53.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:54.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:21:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:21:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:21:54.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:55.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:55.568Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:56.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:56.370Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:56.371Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:21:56.613Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:57.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:57.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:58.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:58.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:21:59.171Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:21:59.408Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:21:59.466Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:21:59.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:21:59.518Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:21:59.518Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:21:59.518Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:21:59.529Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:21:59.548Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:21:59.638Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:00.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:00.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:01.147Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154520234413497, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:22:01.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:01.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:01.147Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:22:01.147Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:22:01.148Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=535.313µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:22:01.325Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:01.469Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:02.302Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:02.617Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:03.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:03.445Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:03.450Z INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type PeriodicPush from 10.0.9.103:8091 2024-05-31T11:22:03.450Z ERRO GOXDCR.P2PManager: PeriodicPushHandler Unable to find opaque 914030592 2024-05-31T11:22:04.262Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:04.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:22:04.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:22:04.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:22:04.501Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:05.158Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:05.584Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:06.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:06.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:06.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:06.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:07.324Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:07.539Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:08.153Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:08.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:09.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:09.463Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:10.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:10.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:11.161Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:11.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:12.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:12.595Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:13.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:13.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:14.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:14.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:22:14.407Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:22:14.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:22:14.407Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:22:14.407Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":126638192,"TotalAlloc":90986437725816,"Sys":327718216,"Lookups":0,"Mallocs":969231653915,"Frees":969230394301,"HeapAlloc":126638192,"HeapSys":308477952,"HeapIdle":160169984,"HeapInuse":148307968,"HeapReleased":62742528,"HeapObjects":1259614,"StackInuse":1900544,"StackSys":1900544,"MSpanInuse":2363520,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191482,"GCSys":7331136,"OtherSys":1611742,"NextGC":224479120,"LastGC":1717154534314551276,"PauseTotalNs":3772549332680,"PauseNs":[125937,127019,128187,108645,208140,134346,201248,147478,131657,130117,134012,133108,126518,297949,116780,139571,118913,297967,130941,130869,133383,345388,124120,138053,128185,128677,200524,295508,126884,124651,146954,114674,128626,133816,134724,131754,134679,140628,295272,133133,138043,133549,120440,129407,123569,126230,173777,134524,235308,161152,117631,125424,129733,127821,130110,122725,299574,157283,125333,106595,129272,135616,136384,170988,132919,134784,136404,161855,125893,149344,126150,281976,132140,295329,744533,130197,132292,119784,207229,146941,131073,113704,130254,129464,133941,136908,127586,125652,189091,125260,132087,156941,120150,212435,135927,287810,121294,133331,141636,125069,156404,126948,135364,127792,122616,130966,110653,140834,136904,132012,129554,127266,304535,6841437,137886,135307,133311,127591,124118,128285,123426,142470,153576,285294,124369,142563,768470,133780,299570,147125,145440,130974,784880,136059,281452,145942,135657,5431339,144717,128976,154506,2987284,130451,131683,330538,132018,7726318,142597,139783,9680074,151618,958567,8175494,137779,4138721,12226545,20471872,4158254,1612395,11240992,6247753,8112920,5742605,8129591,3534317,28375970,12944981,262861,166287,6101686,243943,12503820,12797339,4000796,4076930,16410324,140105,152760,159692,15077573,132033,32962685,16161295,13467089,10015867,141636,17217246,1944681,12560663,10087174,27175589,17813031,7859973,11352371,25834144,11960956,25332742,24565770,24409261,21627200,22141177,10935274,16672527,14319103,11567444,8407322,11543958,5167827,8054693,22901007,12164348,9738598,16125497,128661,27007407,8122667,16176929,5822120,177769,16690365,770453,15639196,23809302,15625464,4087913,4055873,25189699,8087135,17536871,24869952,16424437,9524747,7984663,5054938,15164366,3610921,36431979,5715496,133671,8624451,1413876,7587026,151698,14337869,28530649,9659909,10186917,135207,137051,157460,131148,120215,134686,134418,136588,136556],"PauseEnd":[1717154452384008662,1717154452510728108,1717154452637626042,1717154452759602384,1717154452882431004,1717154453008249459,1717154453132969195,1717154453268239879,1717154453396896242,1717154453524211062,1717154453647066369,1717154453780205544,1717154453911965135,1717154454041173967,1717154454165742786,1717154454298273675,1717154454426295118,1717154454552677964,1717154454680034606,1717154454806511367,1717154454939083379,1717154455083316483,1717154455211004457,1717154455335264392,1717154455465205476,1717154455593935462,1717154455723227202,1717154455851640779,1717154455978607273,1717154456114009379,1717154456242533582,1717154456386428666,1717154456524784901,1717154456654713803,1717154456778595113,1717154456903474416,1717154457037013081,1717154457168936232,1717154457297833418,1717154457424024140,1717154457557254943,1717154457683596341,1717154457807701798,1717154457930499329,1717154458060946558,1717154458184143976,1717154458313954414,1717154458435717134,1717154458561453051,1717154458685870686,1717154458809412681,1717154458934643622,1717154459070689327,1717154459200327076,1717154459315744182,1717154459444318979,1717154459567286177,1717154459690029741,1717154459812407534,1717154459934908076,1717154460086117879,1717154460221445207,1717154460357347524,1717154460492986655,1717154460615953071,1717154460741138390,1717154460866630466,1717154460992019251,1717154461108800121,1717154461233911280,1717154461377285670,1717154461508230110,1717154461633526213,1717154461765182934,1717154461890907813,1717154462016759964,1717154462139599435,1717154462275083201,1717154462396507459,1717154462524290895,1717154462650265263,1717154462779544412,1717154462899737434,1717154463029977789,1717154463163411315,1717154463289670700,1717154463416132096,1717154463538388805,1717154463650585423,1717154463769741028,1717154463892908215,1717154464032369122,1717154464184668257,1717154464317777715,1717154464462357805,1717154464613353887,1717154464750585095,1717154464886509378,1717154465043140535,1717154465193284426,1717154465335679646,1717154465473989725,1717154465609236601,1717154465753897123,1717154465899829567,1717154466038460088,1717154466178915060,1717154466331220027,1717154466483299821,1717154466625328799,1717154466763973516,1717154466912784002,1717154467059251686,1717154467219390199,1717154467388852002,1717154467537699316,1717154467676922403,1717154467813024777,1717154467952129898,1717154468122314101,1717154468285738169,1717154468438676810,1717154468580974808,1717154468732389076,1717154468884196942,1717154469068374548,1717154469277278528,1717154469483112304,1717154469675611493,1717154469863570238,1717154470065737532,1717154470270595743,1717154470481632247,1717154470664501193,1717154470845081853,1717154471049077507,1717154471282309172,1717154471507257839,1717154471700470590,1717154471881233901,1717154472113033805,1717154472422424983,1717154472629652253,1717154472831613259,1717154473027765086,1717154473247532709,1717154473497117243,1717154473697506979,1717154473879924458,1717154474283894546,1717154474851048590,1717154475421410424,1717154475975771270,1717154476559464533,1717154477042727352,1717154477822928067,1717154478346164877,1717154478900252890,1717154479443806394,1717154479921660179,1717154480472793855,1717154480962775471,1717154481565175014,1717154482054220501,1717154482754119858,1717154483312855254,1717154483894471930,1717154484469874924,1717154484956477110,1717154485554946694,1717154486097555758,1717154486773563081,1717154487445329696,1717154488048750146,1717154488672115769,1717154489194525290,1717154489717320669,1717154490230445873,1717154490760473828,1717154491322790529,1717154491816492656,1717154492482458697,1717154493073086194,1717154493658803393,1717154494264012818,1717154494781492234,1717154495322746382,1717154495940713568,1717154496674674134,1717154497493430925,1717154498408071020,1717154499112127480,1717154499783137959,1717154500554594906,1717154501392707757,1717154502129926999,1717154502973971540,1717154503741647109,1717154504548887895,1717154505315707766,1717154506051062419,1717154506769937842,1717154507566923537,1717154508417894921,1717154509259803882,1717154509954611417,1717154510661061140,1717154511302093593,1717154511872040991,1717154512526766203,1717154513132891827,1717154513746696033,1717154514380188534,1717154514902496849,1717154515479760162,1717154515944549595,1717154516582903170,1717154517083653113,1717154517833921451,1717154518435873925,1717154518923540917,1717154519577481419,1717154520099770246,1717154520698021601,1717154521235279712,1717154521817183226,1717154522479224986,1717154523152511711,1717154523769790974,1717154524348329931,1717154524934690733,1717154525546327260,1717154526089024194,1717154526679795566,1717154527202600696,1717154527954565664,1717154528614039967,1717154529103334956,1717154529718985602,1717154530288296456,1717154530817730354,1717154531374988909,1717154531874082970,1717154532608872176,1717154533203655256,1717154533762677663,1717154534314551276,1717154451112627133,1717154451240920472,1717154451378128963,1717154451527271128,1717154451666996349,1717154451803353516,1717154451936315011,1717154452084515464,1717154452244076500],"NumGC":1076727,"NumForcedGC":0,"GCCPUFraction":0.036325065189450945,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437111658,"Frees":437104954},{"Size":16,"Mallocs":6764510613,"Frees":6764148582},{"Size":24,"Mallocs":19132099687,"Frees":19131948639},{"Size":32,"Mallocs":10355274999,"Frees":10355187955},{"Size":48,"Mallocs":1988350752,"Frees":1988302741},{"Size":64,"Mallocs":240453943,"Frees":240421766},{"Size":80,"Mallocs":238718831,"Frees":238703590},{"Size":96,"Mallocs":914677287891,"Frees":914676832045},{"Size":112,"Mallocs":383104333,"Frees":383100367},{"Size":128,"Mallocs":9667674827,"Frees":9667660603},{"Size":144,"Mallocs":190086114,"Frees":190081720},{"Size":160,"Mallocs":71542236,"Frees":71540247},{"Size":176,"Mallocs":5494570,"Frees":5493769},{"Size":192,"Mallocs":32344397,"Frees":32343942},{"Size":208,"Mallocs":68204320,"Frees":68200326},{"Size":224,"Mallocs":3198394,"Frees":3197868},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52359277,"Frees":52331722},{"Size":288,"Mallocs":98502148,"Frees":98480055},{"Size":320,"Mallocs":49949881,"Frees":49946401},{"Size":352,"Mallocs":15126350,"Frees":15125783},{"Size":384,"Mallocs":30408751,"Frees":30408567},{"Size":416,"Mallocs":5922450,"Frees":5919958},{"Size":448,"Mallocs":5902084,"Frees":5897777},{"Size":480,"Mallocs":642393,"Frees":642041},{"Size":512,"Mallocs":39565387,"Frees":39565148},{"Size":576,"Mallocs":21078493,"Frees":21076062},{"Size":640,"Mallocs":1231949,"Frees":1231704},{"Size":704,"Mallocs":18200745,"Frees":18200031},{"Size":768,"Mallocs":6972006,"Frees":6971671},{"Size":896,"Mallocs":7312694,"Frees":7312287},{"Size":1024,"Mallocs":52506155,"Frees":52505952},{"Size":1152,"Mallocs":24945892,"Frees":24945821},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16237176,"Frees":16237147},{"Size":1536,"Mallocs":1798343,"Frees":1798339},{"Size":1792,"Mallocs":12366776,"Frees":12364680},{"Size":2048,"Mallocs":709323,"Frees":709226},{"Size":2304,"Mallocs":22920421,"Frees":22919900},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2035882,"Frees":2035865},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8453803,"Frees":8453783},{"Size":4096,"Mallocs":3534651,"Frees":3534564},{"Size":4864,"Mallocs":8018992,"Frees":8018957},{"Size":5376,"Mallocs":1748262,"Frees":1746718},{"Size":6144,"Mallocs":1308889,"Frees":1308879},{"Size":6528,"Mallocs":6056229,"Frees":6056216},{"Size":6784,"Mallocs":1756561,"Frees":1756487},{"Size":6912,"Mallocs":249627,"Frees":249627},{"Size":8192,"Mallocs":225888,"Frees":225786},{"Size":9472,"Mallocs":1836725,"Frees":1836704},{"Size":9728,"Mallocs":106500,"Frees":106500},{"Size":10240,"Mallocs":1187640,"Frees":1187638},{"Size":10880,"Mallocs":33516,"Frees":33516},{"Size":12288,"Mallocs":272754,"Frees":272750},{"Size":13568,"Mallocs":7812479,"Frees":7812322},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1387721,"Frees":1387690},{"Size":18432,"Mallocs":1189198,"Frees":1189193}]} 2024-05-31T11:22:14.439Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:22:14.464Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:22:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:22:14.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:15.162Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:15.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:16.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:16.356Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:16.369Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:16.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:17.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:17.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:18.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:18.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:19.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:19.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:20.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:20.604Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:21.276Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:21.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:22.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:22.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:23.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154542561724759, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:22:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:22:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:22:23.406Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=20.81212ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:22:23.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:24.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:24.423Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:22:24.424Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:22:24.424Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:22:25.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:25.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:26.307Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:26.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:26.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:26.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:27.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:27.597Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:28.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:28.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:29.306Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:22:29.423Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:22:29.483Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:22:29.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:29.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:22:29.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:22:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:22:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:22:29.499Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:22:30.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:30.556Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154550252655611, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:22:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:22:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:22:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=453.584µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:22:31.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:31.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:32.265Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:32.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:33.148Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:33.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:34.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:34.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:22:34.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:22:34.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:22:34.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:35.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:35.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:36.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:36.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:36.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:36.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:37.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:37.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:38.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:38.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:39.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:39.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:40.334Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:40.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:41.321Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:41.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:42.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:42.650Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:43.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:43.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:44.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:44.415Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:22:44.432Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:22:44.432Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:22:44.432Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:22:44.453Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:22:44.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:22:44.509Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:22:44.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:45.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:45.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:46.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:46.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:46.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:46.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:47.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:47.538Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:48.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:48.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:49.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:49.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:50.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:50.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:51.341Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:51.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:52.310Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:52.638Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:53.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:53.404Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 366, "num_failedckpts": 0, "time_committing": {"count": 366, "max": 1342, "mean": 26.830601092896174, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154572639287810, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 366, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:22:53.404Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:53.405Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:22:53.406Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:22:53.406Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:22:53.407Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=490.911µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:22:53.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:54.191Z ERRO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Pipeline has not been started (or has been disabled) - checkpointing is skipped 2024-05-31T11:22:54.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:22:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:22:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:22:54.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:55.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:55.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:56.124Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Done checkpointing with vb list [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511] totalSize=0 2024-05-31T11:22:56.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:56.394Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:56.394Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:22:56.622Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:57.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:57.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:57.520Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Done checkpointing for replication 2024-05-31T11:22:57.522Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt.disableCkptAndWait took 10m2.803187624s 2024-05-31T11:22:57.544Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:57.544Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:57.544Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:22:57.577Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:57.613Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:57.702Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:57.738Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:57.738Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:22:58.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:58.660Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:22:58.912Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:58.913Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:22:59.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:22:59.407Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:22:59.447Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:22:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:22:59.521Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:22:59.631Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:00.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:00.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:01.148Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154580366998744, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:23:01.148Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:01.148Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:01.148Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:23:01.148Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:23:01.149Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=479.003µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:23:01.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:01.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:02.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:02.694Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:03.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:03.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:04.295Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:04.411Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:23:04.412Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:23:04.412Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:23:04.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:05.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:05.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:06.286Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:06.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:06.389Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:06.613Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:07.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:07.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:08.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:08.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:09.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:09.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:10.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:10.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:11.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:11.594Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:12.253Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:12.714Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:13.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:13.593Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:13.662Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:23:14.375Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:14.439Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:23:14.439Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:23:14.439Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:23:14.440Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:23:14.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:23:14.496Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:23:14.579Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:23:14.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:14.813Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:23:14.813Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:23:15.051Z WARN GOXDCR.Utils: ckmgr.mergeAndPersistBrokenMappingDocsAndCkpts took 16.152434817s 2024-05-31T11:23:15.237Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:15.670Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:16.347Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:16.348Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:16.400Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:16.435Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt took 10m21.71612254s 2024-05-31T11:23:16.435Z INFO GOXDCR.P2PManager: Done handling peer-to-peer push requests from 10.0.9.103:8091 (opaque 115408896 timeTaken 10m21.719092201s) 2024-05-31T11:23:16.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:17.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:17.696Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:18.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:18.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:19.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:19.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:20.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:20.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:21.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:21.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:22.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:22.629Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:23.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:23.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154602644975787, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:23:23.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:23.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:23.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:23:23.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:23:23.395Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=369.924µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:23:23.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:24.160Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:24.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:23:24.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:23:24.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:23:24.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:25.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:25.574Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:26.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:26.627Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:26.637Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:26.647Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:27.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:27.604Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:28.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:28.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:29.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:23:29.448Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:23:29.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:29.472Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:23:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:23:30.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:30.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:31.152Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154610287541081, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:23:31.152Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:31.152Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:31.152Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:23:31.152Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:23:31.153Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=483.953µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:23:31.330Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:31.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:32.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:32.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:33.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:33.462Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:34.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:23:34.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:23:34.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:23:34.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:35.167Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:35.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:36.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:36.393Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:36.393Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:36.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:37.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:37.664Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:38.157Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:38.568Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:39.161Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:39.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:40.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:40.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:41.321Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:41.469Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:42.320Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:42.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:43.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:43.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:44.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:44.408Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:23:44.409Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:23:44.409Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:23:44.409Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:23:44.462Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:23:44.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:44.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:23:44.527Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:23:45.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:45.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:46.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:46.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:46.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:46.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:47.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:47.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:48.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:48.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:49.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:49.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:50.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:50.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:51.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:51.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:52.171Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:52.622Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:53.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154632623074306, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:23:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:23:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:23:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:23:53.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=377.599µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:23:53.443Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:54.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:23:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:23:54.418Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:23:54.568Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:55.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:55.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:56.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:56.398Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:56.398Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:23:56.574Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:57.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:57.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:58.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:58.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:23:59.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:23:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:23:59.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:23:59.515Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:23:59.553Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:23:59.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:00.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:00.611Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154640214895093, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:24:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:24:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:24:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=455.542µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:24:01.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:01.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:02.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:02.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:03.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:03.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:04.156Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:04.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:24:04.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:24:04.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:24:04.621Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:05.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:05.522Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:06.294Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:06.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:06.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:06.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:07.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:07.575Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:08.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:08.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:09.310Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:09.511Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:10.244Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:10.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:11.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:11.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:12.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:12.666Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:13.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:13.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:14.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:14.409Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":187717816,"TotalAlloc":91006678136216,"Sys":327718216,"Lookups":0,"Mallocs":969448659972,"Frees":969446757221,"HeapAlloc":187717816,"HeapSys":308445184,"HeapIdle":97779712,"HeapInuse":210665472,"HeapReleased":66224128,"HeapObjects":1902751,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3584160,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191482,"GCSys":7396672,"OtherSys":1611742,"NextGC":207321408,"LastGC":1717154653962029840,"PauseTotalNs":3774664019923,"PauseNs":[25015485,3160917,146567,10152315,7513213,6758123,12094473,4031562,9697482,21768366,8176816,13735766,175092,27487850,7383963,12976316,16479839,13465049,96684,12779925,12066462,12219780,150023,12328753,139499,16180797,9038381,22896513,20546078,8620106,111901,7732956,144385,139877,7701929,12317694,139536,8025100,137714,8337320,16033075,5396898,9847371,14274453,124850,12124366,153342,12179184,22932969,12158208,140982,147209,4042478,14326496,19694844,717513,12849959,11571799,18150043,7935095,19140218,15440206,8414165,6452656,30120816,1009843,11337699,14832171,31881173,20177245,23966701,4073236,15179239,140356,396456,22149097,11423407,7477820,21246048,7734817,5564306,8109448,24922894,13842589,27182059,2145690,158289,13149751,12577959,9733141,301170,20251783,26446915,6807635,12606449,177498,94648,154011,20294125,10883332,3933742,3269904,150695,20060011,12131983,19746784,158232,10599580,213278,18922872,17016062,14795708,1439181,8698671,143895,8213053,12090902,15341215,12502757,132987,18224106,12229304,16182916,6584009,7450219,13648741,23284394,4115483,152295,17731906,8116360,6023788,17251076,1801229,9507670,8115627,12603270,12490532,121671,9865021,148315,84753,161856,11498966,22299912,169444,132758,12085846,11499658,204433,14952051,15867150,21395436,10410621,22884055,15430959,157639,16531889,22187562,27276782,11252647,18564515,8976954,157104,20630605,7861382,161861,5577953,8715591,20992021,21520053,129307,7971138,11054172,12509284,29435979,11140239,8212825,31128412,22746201,9868450,16206788,171035,1441425,7505850,8135237,8910652,13456283,17997919,19818931,27175589,17813031,7859973,11352371,25834144,11960956,25332742,24565770,24409261,21627200,22141177,10935274,16672527,14319103,11567444,8407322,11543958,5167827,8054693,22901007,12164348,9738598,16125497,128661,27007407,8122667,16176929,5822120,177769,16690365,770453,15639196,23809302,15625464,4087913,4055873,25189699,8087135,17536871,24869952,16424437,9524747,7984663,5054938,15164366,3610921,36431979,5715496,133671,8624451,1413876,7587026,151698,14337869,28530649,9659909,10186917,11086230,11966052,7950846,11468214,17942282,3661783,6274210,159468,16617329],"PauseEnd":[1717154539978782507,1717154540586871257,1717154541065309775,1717154541664040213,1717154542200944807,1717154542887880812,1717154543446888357,1717154543989462946,1717154544595588956,1717154545123373480,1717154545685048630,1717154546240605739,1717154546784465527,1717154547411861932,1717154548054976694,1717154548693772509,1717154549201726146,1717154549779124813,1717154550394293819,1717154550957787017,1717154551604932372,1717154552151353944,1717154552834954487,1717154553338563732,1717154553840041450,1717154554458521001,1717154554966936893,1717154555651517351,1717154556165791289,1717154556730729450,1717154557276605607,1717154557980962876,1717154558576603910,1717154559127924471,1717154559718606438,1717154560235158165,1717154560804707047,1717154561390855482,1717154561914616406,1717154562584633995,1717154563222461465,1717154563790126653,1717154564386926670,1717154564939087123,1717154565498474750,1717154566020193568,1717154566650693280,1717154567189375264,1717154567897249203,1717154568484640476,1717154568967798256,1717154569550745450,1717154570057754683,1717154570637096253,1717154571185394233,1717154571725131228,1717154572328464735,1717154573002807737,1717154573601380647,1717154574158885421,1717154574752868123,1717154575311015931,1717154575843947399,1717154576486824240,1717154577122475209,1717154577969921960,1717154578691316773,1717154579463495528,1717154580295916955,1717154581005685036,1717154581775518690,1717154582626944276,1717154583460210868,1717154584262216802,1717154584987129543,1717154585731839563,1717154586482317367,1717154587219031424,1717154588079212217,1717154588807225322,1717154589566843083,1717154590316551334,1717154591015948642,1717154591800227428,1717154592729994755,1717154593640961486,1717154594303343063,1717154594966293693,1717154595653653428,1717154596346021879,1717154596893863266,1717154597645611484,1717154598370300923,1717154598881543199,1717154599420846400,1717154599936071536,1717154600519401673,1717154601006091795,1717154601587257710,1717154602056608378,1717154602766875864,1717154603339034316,1717154603885711543,1717154604455099561,1717154604944449597,1717154605502124571,1717154605987445647,1717154606650252027,1717154607292520238,1717154608087696607,1717154608634526300,1717154609150184337,1717154609679709294,1717154610183173152,1717154610722047189,1717154611276556336,1717154611879338010,1717154612514613571,1717154613162343148,1717154613745684494,1717154614317673067,1717154614872781249,1717154615427474620,1717154615892186079,1717154616462061665,1717154617092420345,1717154617876124426,1717154618463239826,1717154618930186323,1717154619520800541,1717154620019572794,1717154620613847101,1717154621148247165,1717154621735010452,1717154622265911739,1717154622976734547,1717154623607774339,1717154624115202177,1717154624675885911,1717154625205618756,1717154625756792750,1717154626367688638,1717154626855363911,1717154627384540265,1717154628028614341,1717154628633974927,1717154629103231291,1717154629678492454,1717154630213168996,1717154630755830968,1717154631366716121,1717154631908545514,1717154632431977572,1717154633071400079,1717154633638317937,1717154634096517428,1717154634676718052,1717154635201898739,1717154635813966480,1717154636419741676,1717154636947360220,1717154637566348799,1717154638274618289,1717154638878374221,1717154639462249876,1717154639965735063,1717154640523054484,1717154641020950121,1717154641618611088,1717154642135276605,1717154642812628478,1717154643418173150,1717154643984604225,1717154644552051768,1717154645051099390,1717154645662278854,1717154646220535947,1717154646771429790,1717154647365134501,1717154648081553589,1717154648665412985,1717154649244102324,1717154649794706916,1717154650358580332,1717154650888608451,1717154651503489220,1717154652006196460,1717154652629810488,1717154653378690916,1717154653962029840,1717154498408071020,1717154499112127480,1717154499783137959,1717154500554594906,1717154501392707757,1717154502129926999,1717154502973971540,1717154503741647109,1717154504548887895,1717154505315707766,1717154506051062419,1717154506769937842,1717154507566923537,1717154508417894921,1717154509259803882,1717154509954611417,1717154510661061140,1717154511302093593,1717154511872040991,1717154512526766203,1717154513132891827,1717154513746696033,1717154514380188534,1717154514902496849,1717154515479760162,1717154515944549595,1717154516582903170,1717154517083653113,1717154517833921451,1717154518435873925,1717154518923540917,1717154519577481419,1717154520099770246,1717154520698021601,1717154521235279712,1717154521817183226,1717154522479224986,1717154523152511711,1717154523769790974,1717154524348329931,1717154524934690733,1717154525546327260,1717154526089024194,1717154526679795566,1717154527202600696,1717154527954565664,1717154528614039967,1717154529103334956,1717154529718985602,1717154530288296456,1717154530817730354,1717154531374988909,1717154531874082970,1717154532608872176,1717154533203655256,1717154533762677663,1717154534314551276,1717154534889570281,1717154535460545964,1717154535987198718,1717154536564628426,1717154537133011521,1717154537889710367,1717154538399483663,1717154538920653499,1717154539471460363],"NumGC":1076926,"NumForcedGC":0,"GCCPUFraction":0.03633690268999322,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437274574,"Frees":437267676},{"Size":16,"Mallocs":6767308383,"Frees":6766946125},{"Size":24,"Mallocs":19137405027,"Frees":19137252290},{"Size":32,"Mallocs":10358189201,"Frees":10358102674},{"Size":48,"Mallocs":1989100393,"Frees":1989052235},{"Size":64,"Mallocs":240521025,"Frees":240490369},{"Size":80,"Mallocs":238777394,"Frees":238762151},{"Size":96,"Mallocs":914877495474,"Frees":914876393797},{"Size":112,"Mallocs":383195548,"Frees":383191584},{"Size":128,"Mallocs":9670293608,"Frees":9670278886},{"Size":144,"Mallocs":190171759,"Frees":190167372},{"Size":160,"Mallocs":71554773,"Frees":71552785},{"Size":176,"Mallocs":5496166,"Frees":5495361},{"Size":192,"Mallocs":32353121,"Frees":32352663},{"Size":208,"Mallocs":68219996,"Frees":68215981},{"Size":224,"Mallocs":3198734,"Frees":3198207},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52368487,"Frees":52340917},{"Size":288,"Mallocs":98599058,"Frees":98580503},{"Size":320,"Mallocs":49956463,"Frees":49952996},{"Size":352,"Mallocs":15134089,"Frees":15133536},{"Size":384,"Mallocs":30417194,"Frees":30417008},{"Size":416,"Mallocs":5923813,"Frees":5921333},{"Size":448,"Mallocs":5902111,"Frees":5897804},{"Size":480,"Mallocs":642393,"Frees":642041},{"Size":512,"Mallocs":39577187,"Frees":39576922},{"Size":576,"Mallocs":21100330,"Frees":21097891},{"Size":640,"Mallocs":1232208,"Frees":1231963},{"Size":704,"Mallocs":18206193,"Frees":18205479},{"Size":768,"Mallocs":6973794,"Frees":6973459},{"Size":896,"Mallocs":7314656,"Frees":7314245},{"Size":1024,"Mallocs":52519823,"Frees":52519599},{"Size":1152,"Mallocs":24968970,"Frees":24968885},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16241581,"Frees":16241552},{"Size":1536,"Mallocs":1798842,"Frees":1798838},{"Size":1792,"Mallocs":12371091,"Frees":12368991},{"Size":2048,"Mallocs":709505,"Frees":709406},{"Size":2304,"Mallocs":22942927,"Frees":22942392},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2036277,"Frees":2036258},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8456223,"Frees":8456201},{"Size":4096,"Mallocs":3535426,"Frees":3535330},{"Size":4864,"Mallocs":8026901,"Frees":8026858},{"Size":5376,"Mallocs":1749354,"Frees":1747810},{"Size":6144,"Mallocs":1311985,"Frees":1311975},{"Size":6528,"Mallocs":6058016,"Frees":6058003},{"Size":6784,"Mallocs":1757194,"Frees":1757118},{"Size":6912,"Mallocs":249687,"Frees":249687},{"Size":8192,"Mallocs":225946,"Frees":225843},{"Size":9472,"Mallocs":1837340,"Frees":1837315},{"Size":9728,"Mallocs":106549,"Frees":106547},{"Size":10240,"Mallocs":1187880,"Frees":1187878},{"Size":10880,"Mallocs":33528,"Frees":33528},{"Size":12288,"Mallocs":272823,"Frees":272819},{"Size":13568,"Mallocs":7814899,"Frees":7814738},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1388116,"Frees":1388084},{"Size":18432,"Mallocs":1189634,"Frees":1189625}]} 2024-05-31T11:24:14.410Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:24:14.411Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:24:14.411Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:24:14.427Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:24:14.494Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:24:14.533Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:24:14.619Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:24:14.623Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:15.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:15.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:16.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:16.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:16.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:16.479Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:17.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:17.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:18.249Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:18.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:19.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:19.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:20.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:20.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:21.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:21.593Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:22.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:22.659Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:23.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154662659848635, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:24:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:24:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:24:23.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=374.412µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:24:23.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:24.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:24.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:24:24.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:24:24.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:24:24.611Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:25.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:25.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:26.383Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:26.384Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:26.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:26.678Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:27.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:27.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:28.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:28.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:29.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:29.406Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:24:29.485Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:24:29.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:24:29.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:29.585Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:24:30.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:30.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:31.153Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154670266620757, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:24:31.153Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:31.153Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:31.153Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:24:31.153Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:24:31.153Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=501.313µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:24:31.423Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:31.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:32.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:32.639Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:33.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:33.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:34.176Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:34.413Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:24:34.413Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:24:34.413Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:24:34.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:35.249Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:35.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:36.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:36.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:36.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:36.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:37.314Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:37.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:38.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:38.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:39.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:39.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:40.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:40.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:41.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:41.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:42.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:42.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:43.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:43.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:44.302Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:44.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:24:44.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:24:44.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:24:44.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:24:44.436Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:24:44.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:44.492Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:24:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:24:45.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:45.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:46.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:46.361Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:46.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:46.609Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:47.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:47.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:48.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:48.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:49.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:49.556Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:50.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:50.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:51.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:51.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:52.353Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:52.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:53.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:53.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154692556914078, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:24:53.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:53.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:24:53.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:24:53.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:24:53.396Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=449.111µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:24:53.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:54.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:54.401Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:24:54.401Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:24:54.401Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:24:54.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:55.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:55.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:56.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:56.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:56.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:24:56.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:57.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:57.627Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:58.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:58.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:24:59.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:24:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:24:59.454Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:24:59.491Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:24:59.501Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:24:59.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:00.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:00.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154700229501758, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:25:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:25:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:25:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=497.269µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:25:01.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:01.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:02.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:02.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:03.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:03.511Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:04.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:25:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:25:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:25:04.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:05.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:05.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:06.262Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:06.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:06.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:06.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:07.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:07.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:08.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:08.595Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:10.105Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:10.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:11.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:11.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:12.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:12.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:13.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:13.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:14.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:14.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:14.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:25:14.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:25:14.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:25:14.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:25:14.410Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:25:14.428Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:25:14.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:25:15.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:15.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:16.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:16.361Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:16.362Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:16.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:17.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:17.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:18.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:18.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:19.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:19.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:20.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:20.375Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:21.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:21.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:22.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:22.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:23.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:23.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:23.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154723358555532, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:25:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:25:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:25:23.389Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=346.568µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:25:24.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:24.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:24.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:25:24.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:25:24.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:25:25.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:25.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:26.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:26.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:26.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:26.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:27.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:27.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:28.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:28.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:29.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:29.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:29.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:25:29.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:25:29.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:25:29.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:25:29.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:25:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:25:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:25:29.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:25:30.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:30.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:31.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:31.145Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154731075484736, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:25:31.145Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:31.145Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:31.145Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:25:31.146Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:25:31.146Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=466.739µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:25:31.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:32.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:32.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:33.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:33.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:34.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:34.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:25:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:25:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:25:35.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:35.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:36.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:36.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:36.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:37.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:38.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:38.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:39.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:39.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:40.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:40.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:41.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:41.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:42.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:42.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:43.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:43.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:44.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:44.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:44.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:25:44.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:25:44.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:25:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:25:44.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:25:44.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:25:44.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:25:44.507Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:25:44.507Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:25:44.508Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:25:44.510Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:25:45.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:45.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:46.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:46.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:46.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:46.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:47.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:47.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:48.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:48.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:49.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:49.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:50.090Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:50.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:51.089Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:51.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:52.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:52.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:53.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:53.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154753364323743, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:25:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:25:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:25:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:25:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=408.492µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:25:54.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:54.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:25:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:25:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:25:55.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:55.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:56.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:56.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:56.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:25:56.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:57.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:57.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:58.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:58.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:59.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:25:59.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:25:59.399Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:25:59.415Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:25:59.430Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:25:59.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:25:59.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:25:59.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:25:59.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:25:59.508Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:26:00.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:00.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:01.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154761074830389, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:26:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:26:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:26:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=498.592µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:26:01.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:02.089Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:02.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:03.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:03.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:04.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:04.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:26:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:26:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:26:05.092Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:05.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:06.097Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:06.330Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:06.330Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:06.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:07.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:07.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:08.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:08.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:09.109Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:09.416Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:10.089Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:11.120Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:11.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:12.113Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:12.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:13.106Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:13.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:14.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:14.406Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":161296128,"TotalAlloc":91058209565400,"Sys":327718216,"Lookups":0,"Mallocs":969991757156,"Frees":969990116461,"HeapAlloc":161296128,"HeapSys":308510720,"HeapIdle":124780544,"HeapInuse":183730176,"HeapReleased":65290240,"HeapObjects":1640695,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3052480,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191482,"GCSys":7331136,"OtherSys":1611742,"NextGC":209329560,"LastGC":1717154774100409718,"PauseTotalNs":3775961372882,"PauseNs":[173609,126647,140982,125318,129670,12126512,133084,109778,104986,121586,144233,136480,131614,135396,128820,126983,133320,131932,295033,128001,134803,112659,131413,116376,139408,299434,174959,133935,145103,127698,139868,135491,310565,133448,143509,115284,115001,151690,124051,342337,130573,137894,125436,115528,115644,107341,155469,155258,139893,156107,130880,133506,115556,110233,146272,134920,144615,782090,137618,104992,291384,124631,132784,130431,101512,132267,164185,115710,117145,118042,170168,126055,121880,112936,208456,136769,140254,128823,129207,139362,201657,134025,136094,122240,127989,131354,125032,137058,132524,142669,126233,134854,121495,134936,150909,135794,166221,120211,135987,285298,124857,131130,134796,127794,133222,134887,124434,133816,138785,289071,313883,132982,139484,130581,138171,653038,127992,136812,138348,112049,457324,122501,141733,312605,132215,125805,134568,293939,155914,126887,116777,127170,119195,133972,137261,174778,124588,119364,149244,291742,109010,108913,303837,168069,133971,127916,124899,124880,115254,115021,129127,111754,323558,127808,107260,129925,124655,115630,129269,127816,130833,142047,129007,124220,131232,134208,142820,111747,229426,138464,133532,141806,135369,125585,134822,203259,127871,124431,136431,129924,149600,145080,124005,214378,129198,128403,139580,179387,136129,134477,152282,126212,131349,131069,161540,142596,290545,132974,112168,112976,109994,131491,125238,146259,140278,170313,128631,127289,131760,124262,165272,134238,299126,114942,131099,124104,194762,92086,146035,183812,142083,306203,129357,130219,148509,144177,148123,146421,112471,128796,291582,129903,151674,109834,299751,124115,129510,107592,120080,12324786,126407,115112,163947,114038,6519637,9304013,3234376,140497,11463995,337556,3842660,457743,4162673,133295,8169759,184766],"PauseEnd":[1717154772998823043,1717154773224130163,1717154773445374220,1717154773646563158,1717154773825572462,1717154774100409718,1717154738898287240,1717154739019128664,1717154739137746514,1717154739269567713,1717154739395608426,1717154739528444382,1717154739658599418,1717154739781491203,1717154739904974496,1717154740034563334,1717154740153848889,1717154740277870304,1717154740407119211,1717154740540067977,1717154740667194171,1717154740787765960,1717154740904506815,1717154741021239096,1717154741138732953,1717154741268197317,1717154741396123899,1717154741520744995,1717154741644881299,1717154741773069514,1717154741900952602,1717154742032079612,1717154742162509361,1717154742290038598,1717154742413647064,1717154742559975627,1717154742688667151,1717154742814888776,1717154742932503616,1717154743065487912,1717154743197374494,1717154743319563989,1717154743440924018,1717154743560996472,1717154743679643084,1717154743808781347,1717154743931278355,1717154744087899441,1717154744235003874,1717154744383093399,1717154744523238357,1717154744657802191,1717154744783138862,1717154744907453198,1717154745039735463,1717154745177110903,1717154745300992475,1717154745428001114,1717154745557913136,1717154745687017937,1717154745812101843,1717154745936468174,1717154746082470421,1717154746215154156,1717154746350952747,1717154746486954581,1717154746615809246,1717154746742485487,1717154746863468459,1717154746986537717,1717154747110707020,1717154747232712220,1717154747372546304,1717154747507672285,1717154747637206212,1717154747770484536,1717154747893151942,1717154748016620242,1717154748142650765,1717154748267325705,1717154748391932610,1717154748523029967,1717154748653757479,1717154748780732376,1717154748909619813,1717154749046575649,1717154749178423242,1717154749300729588,1717154749419135184,1717154749547642924,1717154749667637012,1717154749790246150,1717154749919599625,1717154750058245713,1717154750186030170,1717154750307392872,1717154750418912967,1717154750541541747,1717154750667631461,1717154750792979576,1717154750923059749,1717154751058616487,1717154751190378981,1717154751311353641,1717154751429142255,1717154751549120986,1717154751671728496,1717154751800116130,1717154751918084802,1717154752043030836,1717154752186414630,1717154752318036018,1717154752446969669,1717154752577470048,1717154752699770681,1717154752840841318,1717154752976548902,1717154753103252790,1717154753232054816,1717154753376491765,1717154753507543512,1717154753620479042,1717154753744063410,1717154753867174839,1717154753990530606,1717154754115796402,1717154754238381096,1717154754375241888,1717154754506291634,1717154754634772779,1717154754753922002,1717154754875736496,1717154755005198772,1717154755124111179,1717154755248136499,1717154755380192777,1717154755517365642,1717154755642965351,1717154755766563224,1717154755888441783,1717154756001799295,1717154756122285202,1717154756257146960,1717154756387145690,1717154756517249020,1717154756640081011,1717154756763081576,1717154756892567845,1717154757020824490,1717154757143534014,1717154757270632482,1717154757396308995,1717154757549438242,1717154757689023380,1717154757816588794,1717154757945308710,1717154758091141841,1717154758231416905,1717154758371958736,1717154758507058107,1717154758635833180,1717154758764162990,1717154758890439433,1717154759013304946,1717154759135813602,1717154759264606734,1717154759399646389,1717154759536011705,1717154759660178024,1717154759784829721,1717154759912484860,1717154760036533759,1717154760170542414,1717154760300405153,1717154760415438169,1717154760543144161,1717154760666351982,1717154760788834190,1717154760915848416,1717154761043639740,1717154761178317966,1717154761305108056,1717154761424293206,1717154761545924208,1717154761673092500,1717154761793841860,1717154761915103866,1717154762045414023,1717154762164411095,1717154762299358737,1717154762424814806,1717154762554304964,1717154762677194147,1717154762799968319,1717154762921601547,1717154763047474969,1717154763166692021,1717154763289578576,1717154763408621948,1717154763537282275,1717154763650599266,1717154763772880681,1717154763898554550,1717154764026102974,1717154764170363848,1717154764327214505,1717154764487847089,1717154764634282171,1717154764776238074,1717154764914443601,1717154765046307912,1717154765192958965,1717154765321735389,1717154765459765209,1717154765611471044,1717154765778943372,1717154765927336773,1717154766097061501,1717154766254198955,1717154766407177806,1717154766562603603,1717154766694888382,1717154766828170480,1717154766961223307,1717154767116494837,1717154767267981915,1717154767435209432,1717154767607101234,1717154767753478037,1717154767892494449,1717154768029278318,1717154768169918348,1717154768320927789,1717154768471430204,1717154768615838600,1717154768752947890,1717154768886166535,1717154769055152480,1717154769247726727,1717154769463545183,1717154769641310905,1717154769835752621,1717154770023001027,1717154770221758925,1717154770447751288,1717154770660375185,1717154770847161386,1717154771035572473,1717154771262813946,1717154771467080295,1717154771655804539,1717154771835399525,1717154772053362242,1717154772278107675,1717154772552029682,1717154772809505330],"NumGC":1077510,"NumForcedGC":0,"GCCPUFraction":0.036345550963791225,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437384187,"Frees":437377503},{"Size":16,"Mallocs":6769110750,"Frees":6768749351},{"Size":24,"Mallocs":19142555482,"Frees":19142395114},{"Size":32,"Mallocs":10360952564,"Frees":10360859868},{"Size":48,"Mallocs":1989651856,"Frees":1989602715},{"Size":64,"Mallocs":240564774,"Frees":240534126},{"Size":80,"Mallocs":238818357,"Frees":238803115},{"Size":96,"Mallocs":915405771632,"Frees":915404951188},{"Size":112,"Mallocs":383286733,"Frees":383282769},{"Size":128,"Mallocs":9672919884,"Frees":9672899464},{"Size":144,"Mallocs":190215384,"Frees":190210998},{"Size":160,"Mallocs":71565255,"Frees":71563267},{"Size":176,"Mallocs":5496754,"Frees":5495954},{"Size":192,"Mallocs":32362246,"Frees":32361786},{"Size":208,"Mallocs":68236262,"Frees":68232281},{"Size":224,"Mallocs":3199055,"Frees":3198529},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52377721,"Frees":52350157},{"Size":288,"Mallocs":98605319,"Frees":98586808},{"Size":320,"Mallocs":49963032,"Frees":49959565},{"Size":352,"Mallocs":15137118,"Frees":15136579},{"Size":384,"Mallocs":30426048,"Frees":30425859},{"Size":416,"Mallocs":5925168,"Frees":5922690},{"Size":448,"Mallocs":5902137,"Frees":5897830},{"Size":480,"Mallocs":642393,"Frees":642041},{"Size":512,"Mallocs":39588211,"Frees":39587979},{"Size":576,"Mallocs":21101678,"Frees":21099245},{"Size":640,"Mallocs":1232463,"Frees":1232218},{"Size":704,"Mallocs":18210612,"Frees":18209898},{"Size":768,"Mallocs":6975939,"Frees":6975601},{"Size":896,"Mallocs":7316731,"Frees":7316322},{"Size":1024,"Mallocs":52533443,"Frees":52533234},{"Size":1152,"Mallocs":24971553,"Frees":24971480},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16245986,"Frees":16245957},{"Size":1536,"Mallocs":1799333,"Frees":1799329},{"Size":1792,"Mallocs":12374856,"Frees":12372755},{"Size":2048,"Mallocs":709691,"Frees":709595},{"Size":2304,"Mallocs":22944937,"Frees":22944414},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2036667,"Frees":2036650},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8459124,"Frees":8459099},{"Size":4096,"Mallocs":3536183,"Frees":3536099},{"Size":4864,"Mallocs":8027639,"Frees":8027604},{"Size":5376,"Mallocs":1749422,"Frees":1747878},{"Size":6144,"Mallocs":1312009,"Frees":1311999},{"Size":6528,"Mallocs":6060157,"Frees":6060141},{"Size":6784,"Mallocs":1757954,"Frees":1757878},{"Size":6912,"Mallocs":249747,"Frees":249747},{"Size":8192,"Mallocs":225994,"Frees":225892},{"Size":9472,"Mallocs":1837957,"Frees":1837936},{"Size":9728,"Mallocs":106595,"Frees":106595},{"Size":10240,"Mallocs":1188120,"Frees":1188118},{"Size":10880,"Mallocs":33538,"Frees":33538},{"Size":12288,"Mallocs":272893,"Frees":272889},{"Size":13568,"Mallocs":7817800,"Frees":7817648},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1388508,"Frees":1388477},{"Size":18432,"Mallocs":1190070,"Frees":1190065}]} 2024-05-31T11:26:14.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:26:14.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:26:14.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:26:14.407Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:26:14.443Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:26:14.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:14.480Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:26:14.491Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:26:14.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:26:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:26:14.500Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:26:14.502Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:26:15.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:15.454Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:16.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:16.392Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:16.392Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:16.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:17.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:17.716Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:18.313Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:18.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:19.176Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:19.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:20.155Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:20.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:21.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:21.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:22.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:22.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:23.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:23.404Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154782477689206, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:26:23.404Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:23.404Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:23.404Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:26:23.405Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:26:23.405Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=365.458µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:26:23.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:24.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:26:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:26:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:26:24.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:25.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:25.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:26.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:26.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:26.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:26.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:27.233Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:27.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:28.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:28.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:29.170Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:29.399Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:26:29.447Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:26:29.491Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:26:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:26:29.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:30.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:30.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154790168881888, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:26:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:31.135Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:26:31.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:26:31.149Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=12.893745ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:26:31.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:31.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:32.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:32.642Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:33.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:33.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:34.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:34.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:26:34.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:26:34.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:26:34.520Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:35.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:35.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:36.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:36.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:36.353Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:36.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:37.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:37.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:38.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:38.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:39.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:39.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:40.244Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:40.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:41.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:41.627Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:42.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:42.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:43.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:43.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:44.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:44.404Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:26:44.461Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:26:44.461Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:26:44.461Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:26:44.467Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:26:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:26:44.502Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:26:44.502Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:26:44.502Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:26:44.504Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:26:44.504Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:26:44.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:45.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:45.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:46.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:46.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:46.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:46.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:47.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:47.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:48.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:48.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:49.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:49.520Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:50.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:50.539Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:51.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:51.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:52.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:52.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:53.309Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:53.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154812570743578, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:26:53.388Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:53.388Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:26:53.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:26:53.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:26:53.388Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=462.772µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:26:53.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:54.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:54.419Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:26:54.419Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:26:54.419Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:26:54.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:55.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:55.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:56.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:56.387Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:56.403Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:26:56.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:57.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:57.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:58.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:58.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:26:59.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:26:59.416Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:26:59.450Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:26:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:26:59.497Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:26:59.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:26:59.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:26:59.500Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:26:59.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:26:59.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:00.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:00.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:01.134Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154820204694604, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:27:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:27:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:27:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=489.941µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:27:01.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:01.597Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:02.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:02.447Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:03.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:03.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:04.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:04.418Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:27:04.419Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:27:04.419Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:27:04.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:05.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:05.503Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:06.153Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:06.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:06.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:06.556Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:07.176Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:07.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:08.294Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:08.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:09.309Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:09.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:10.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:10.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:11.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:11.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:12.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:12.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:13.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:13.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:14.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:14.398Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:27:14.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:27:14.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:27:14.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:27:14.442Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:27:14.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:27:14.498Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:27:14.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:27:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:27:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:27:14.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:27:14.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:15.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:15.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:16.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:16.384Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:16.385Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:16.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:17.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:17.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:18.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:18.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:19.285Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:19.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:20.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:20.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:21.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:21.464Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:22.317Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:22.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:23.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154842482241356, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:27:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:27:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:27:23.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=379.466µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:27:23.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:24.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:24.409Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:27:24.409Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:27:24.409Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:27:24.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:25.167Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:25.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:26.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:26.381Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:26.455Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:26.633Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:27.265Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:27.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:28.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:28.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:29.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:29.430Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:27:29.478Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:27:29.523Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:27:29.538Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:27:29.539Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:27:29.540Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:27:29.559Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:27:29.576Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:27:29.621Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:30.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:30.534Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154850193949884, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:27:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:27:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:27:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=500.771µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:27:31.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:31.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:32.253Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:32.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:33.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:33.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:34.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:34.405Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:27:34.405Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:27:34.405Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:27:34.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:35.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:35.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:36.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:36.375Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:36.375Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:37.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:37.332Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:37.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:38.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:38.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:39.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:39.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:40.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:40.439Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:41.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:41.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:42.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:42.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:43.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:43.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:44.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:27:44.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:27:44.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:27:44.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:27:44.445Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:27:44.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:27:44.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:44.513Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:27:44.513Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:27:44.513Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:27:44.515Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:27:44.532Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:27:45.285Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:45.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:46.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:46.377Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:46.377Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:46.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:47.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:47.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:48.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:48.431Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:49.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:49.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:50.338Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:50.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:51.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:51.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:52.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:52.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:53.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154872577729334, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:27:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:27:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:27:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:27:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=375.979µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:27:53.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:54.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:27:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:27:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:27:54.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:55.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:55.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:56.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:56.389Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:56.389Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:27:56.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:57.176Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:57.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:58.325Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:58.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:27:59.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:27:59.417Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:27:59.475Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:27:59.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:27:59.510Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:27:59.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:00.165Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:00.612Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:01.141Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154880165807918, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:28:01.141Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:01.141Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:01.142Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:28:01.142Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:28:01.143Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=1.54881ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:28:01.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:01.522Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:02.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:02.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:03.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:03.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:04.276Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:04.407Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:28:04.407Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:28:04.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:28:04.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:05.302Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:05.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:06.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:06.418Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:06.418Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:06.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:07.140Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:07.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:08.327Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:08.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:09.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:09.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:10.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:10.617Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:11.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:11.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:12.306Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:12.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:13.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:13.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:14.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:14.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:28:14.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:28:14.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:28:14.422Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":170346944,"TotalAlloc":91079183592112,"Sys":327718216,"Lookups":0,"Mallocs":970216525794,"Frees":970214791209,"HeapAlloc":170346944,"HeapSys":308445184,"HeapIdle":115236864,"HeapInuse":193208320,"HeapReleased":73203712,"HeapObjects":1734585,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3230880,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191482,"GCSys":7396672,"OtherSys":1611742,"NextGC":197119376,"LastGC":1717154893995764539,"PauseTotalNs":3778329933315,"PauseNs":[173609,126647,140982,125318,129670,12126512,19724994,6499051,154594,14747526,12843906,4280196,20321033,8070415,16735506,23107892,20095238,18024117,2827271,138556,2392511,20960020,129115,16086685,138956,12129454,15216676,7756218,20445220,2434723,8007587,4121078,9154949,16797744,19826340,4104100,92121,2841985,13194817,19740640,16157520,20484957,19300134,28403212,13913776,8208677,16551269,13302220,4336770,12269992,14302417,18336751,12869748,4055806,7943780,9309396,8103428,9087645,11684875,16140902,10650695,4561358,22933327,22947364,4113339,18911363,24700131,4009864,132713,17747302,11205012,14528652,11915010,135871,13938957,8015210,2962153,17682933,8953989,13956573,144010,4836853,21648328,4997757,20853009,3957414,10224330,14399230,23319640,137332,10082563,12055705,15623349,20088067,172516,24331177,4349984,146961,23711681,10657676,9885987,8100179,8207494,12821809,20612519,6722989,17564545,318564,21139903,4098683,16163933,154142,15964836,14182344,142842,10569502,136108,24403292,23261149,7270986,6523269,19140209,150880,17211364,130158,8123762,3930286,13964639,18744578,4808089,10359679,5748014,12165951,123714,17258949,172887,15000541,11922282,23548597,21544793,16189081,12455362,8036652,8846115,12144400,20247670,156362,20474551,141253,8126735,137739,8148771,8021878,2628089,14310868,35849914,15753402,306829,17858661,10022228,7960779,9400684,7983027,22472546,4114167,24700867,7989135,15644732,12564820,12188241,162581,19280378,6985096,19523733,4082341,284924,7665945,140514,23051264,15844499,8089036,146696,150217,19016753,10966535,12183997,121256,25940253,8422719,12217128,144284,3491808,16727164,7953420,3874131,10232247,14992390,11438692,12170490,18897201,12694326,12075633,21596488,815394,96916,8139712,20140019,3618786,12927650,740183,11032073,9530143,158678,11954336,156580,30910506,19036217,3503086,22311168,8110935,142083,306203,129357,130219,148509,144177,148123,146421,112471,128796,291582,129903,151674,109834,299751,124115,129510,107592,120080,12324786,126407,115112,163947,114038,6519637,9304013,3234376,140497,11463995,337556,3842660,457743,4162673,133295,8169759,184766],"PauseEnd":[1717154772998823043,1717154773224130163,1717154773445374220,1717154773646563158,1717154773825572462,1717154774100409718,1717154774707462212,1717154775178967977,1717154775752297163,1717154776391618476,1717154776930717383,1717154777626157081,1717154778357105176,1717154778903865789,1717154779461742344,1717154779971893999,1717154780526290242,1717154781064422031,1717154781646559869,1717154782143321879,1717154782737001124,1717154783384131396,1717154783899174745,1717154784457557714,1717154784908311475,1717154785475314207,1717154786058232239,1717154786649949681,1717154787116728640,1717154787687166770,1717154788336493147,1717154788851400542,1717154789378063574,1717154789919032104,1717154790437404388,1717154790895899721,1717154791502921813,1717154791956935725,1717154792500358683,1717154793215376029,1717154793810831642,1717154794349725393,1717154794860442913,1717154795383015010,1717154795870099195,1717154796452875752,1717154796959807534,1717154797504234645,1717154798178945863,1717154798734262867,1717154799240289621,1717154799782027143,1717154800295584125,1717154800888761642,1717154801574448662,1717154802064075296,1717154802697426958,1717154803346662732,1717154803860176743,1717154804393376318,1717154804886670905,1717154805427014140,1717154806018200179,1717154806652912305,1717154807109630605,1717154807779051983,1717154808372918909,1717154808890734562,1717154809391019660,1717154809880463898,1717154810415010972,1717154810925521006,1717154811466651391,1717154811972370021,1717154812534642808,1717154813259287961,1717154813805594540,1717154814370914928,1717154814861712538,1717154815416969758,1717154815910686604,1717154816507901599,1717154817037110932,1717154817641320825,1717154818353182032,1717154818825159162,1717154819356805927,1717154819875774962,1717154820419361334,1717154820933276808,1717154821562451082,1717154822132105059,1717154822717485732,1717154823379051428,1717154823892750546,1717154824418027607,1717154824912691767,1717154825391468446,1717154825900845360,1717154826450923615,1717154826973309842,1717154827527623958,1717154828194469417,1717154828726052645,1717154829259244662,1717154829761201642,1717154830266686386,1717154830799357361,1717154831360537681,1717154831882818865,1717154832441708201,1717154833073490056,1717154833704622479,1717154834251335724,1717154834798516301,1717154835317682658,1717154835843782958,1717154836383102839,1717154836869219004,1717154837440426341,1717154838099440683,1717154838681358509,1717154839147171468,1717154839655569374,1717154840100937144,1717154840641545546,1717154841139551135,1717154841671184290,1717154842175041069,1717154842727561012,1717154843383864387,1717154843886632050,1717154844429368658,1717154844902530707,1717154845466803650,1717154845950928678,1717154846631327277,1717154847202123530,1717154847813397312,1717154848453315603,1717154848968439097,1717154849523039672,1717154850012899379,1717154850583349569,1717154851057161744,1717154851676027309,1717154852182543323,1717154852841653826,1717154853425577302,1717154853951016386,1717154854481389464,1717154854963140475,1717154855500460805,1717154855950401158,1717154856524829521,1717154857214903695,1717154857913257064,1717154858536912489,1717154859070882690,1717154859634708082,1717154860100007122,1717154860599016180,1717154861093081392,1717154861699979455,1717154862189272534,1717154862850914354,1717154863505508790,1717154864072556795,1717154864676276421,1717154865259569924,1717154865807450922,1717154866374574354,1717154866890020625,1717154867432893132,1717154868112732183,1717154868660828692,1717154869191206040,1717154869716787559,1717154870272952650,1717154870799668052,1717154871390312109,1717154871904283367,1717154872459503992,1717154873118798700,1717154873746754916,1717154874325776740,1717154874813525189,1717154875368462575,1717154875894633762,1717154876472614269,1717154876944867932,1717154877514713700,1717154878265941277,1717154878821986229,1717154879416734820,1717154879957805337,1717154880543321602,1717154881068730892,1717154881706229060,1717154882235458547,1717154882853557066,1717154883596354803,1717154884163182411,1717154884734168704,1717154885264429208,1717154885822262128,1717154886417471507,1717154886952326361,1717154887523738453,1717154888260188114,1717154888853175304,1717154889396443239,1717154889954461928,1717154890580904143,1717154891065322867,1717154891702493900,1717154892216918482,1717154892808453022,1717154893460557399,1717154893995764539,1717154766562603603,1717154766694888382,1717154766828170480,1717154766961223307,1717154767116494837,1717154767267981915,1717154767435209432,1717154767607101234,1717154767753478037,1717154767892494449,1717154768029278318,1717154768169918348,1717154768320927789,1717154768471430204,1717154768615838600,1717154768752947890,1717154768886166535,1717154769055152480,1717154769247726727,1717154769463545183,1717154769641310905,1717154769835752621,1717154770023001027,1717154770221758925,1717154770447751288,1717154770660375185,1717154770847161386,1717154771035572473,1717154771262813946,1717154771467080295,1717154771655804539,1717154771835399525,1717154772053362242,1717154772278107675,1717154772552029682,1717154772809505330],"NumGC":1077724,"NumForcedGC":0,"GCCPUFraction":0.036356547987659416,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437492712,"Frees":437486099},{"Size":16,"Mallocs":6770863037,"Frees":6770501948},{"Size":24,"Mallocs":19147646392,"Frees":19147487257},{"Size":32,"Mallocs":10363708124,"Frees":10363616277},{"Size":48,"Mallocs":1990201167,"Frees":1990152133},{"Size":64,"Mallocs":240608141,"Frees":240577528},{"Size":80,"Mallocs":238859423,"Frees":238844182},{"Size":96,"Mallocs":915615848748,"Frees":915614931014},{"Size":112,"Mallocs":383377926,"Frees":383373958},{"Size":128,"Mallocs":9675539530,"Frees":9675519841},{"Size":144,"Mallocs":190259018,"Frees":190254632},{"Size":160,"Mallocs":71575728,"Frees":71573740},{"Size":176,"Mallocs":5497320,"Frees":5496517},{"Size":192,"Mallocs":32371235,"Frees":32370792},{"Size":208,"Mallocs":68252305,"Frees":68248320},{"Size":224,"Mallocs":3199381,"Frees":3198855},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52386939,"Frees":52359383},{"Size":288,"Mallocs":98611377,"Frees":98592858},{"Size":320,"Mallocs":49969601,"Frees":49966137},{"Size":352,"Mallocs":15139892,"Frees":15139352},{"Size":384,"Mallocs":30434749,"Frees":30434577},{"Size":416,"Mallocs":5926558,"Frees":5924080},{"Size":448,"Mallocs":5902163,"Frees":5897856},{"Size":480,"Mallocs":642393,"Frees":642041},{"Size":512,"Mallocs":39598915,"Frees":39598666},{"Size":576,"Mallocs":21103032,"Frees":21100599},{"Size":640,"Mallocs":1232722,"Frees":1232477},{"Size":704,"Mallocs":18215043,"Frees":18214329},{"Size":768,"Mallocs":6977900,"Frees":6977574},{"Size":896,"Mallocs":7318764,"Frees":7318360},{"Size":1024,"Mallocs":52547085,"Frees":52546876},{"Size":1152,"Mallocs":24974165,"Frees":24974090},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16250390,"Frees":16250361},{"Size":1536,"Mallocs":1799837,"Frees":1799833},{"Size":1792,"Mallocs":12378386,"Frees":12376302},{"Size":2048,"Mallocs":709868,"Frees":709770},{"Size":2304,"Mallocs":22946972,"Frees":22946447},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2037063,"Frees":2037046},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8461771,"Frees":8461763},{"Size":4096,"Mallocs":3536953,"Frees":3536866},{"Size":4864,"Mallocs":8028394,"Frees":8028357},{"Size":5376,"Mallocs":1749490,"Frees":1747946},{"Size":6144,"Mallocs":1312033,"Frees":1312023},{"Size":6528,"Mallocs":6062112,"Frees":6062108},{"Size":6784,"Mallocs":1758646,"Frees":1758575},{"Size":6912,"Mallocs":249807,"Frees":249807},{"Size":8192,"Mallocs":226043,"Frees":225941},{"Size":9472,"Mallocs":1838584,"Frees":1838561},{"Size":9728,"Mallocs":106643,"Frees":106643},{"Size":10240,"Mallocs":1188359,"Frees":1188358},{"Size":10880,"Mallocs":33550,"Frees":33550},{"Size":12288,"Mallocs":272961,"Frees":272957},{"Size":13568,"Mallocs":7820447,"Frees":7820298},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1388900,"Frees":1388869},{"Size":18432,"Mallocs":1190512,"Frees":1190504}]} 2024-05-31T11:28:14.406Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:28:14.493Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:28:14.494Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:28:14.634Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:28:14.638Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:15.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:15.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:16.267Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:16.358Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:16.358Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:16.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:17.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:17.584Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:18.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:18.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:19.317Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:19.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:20.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:20.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:21.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:21.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:22.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:22.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:23.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154902615322641, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:28:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:23.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:28:23.408Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:28:23.409Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=481.086µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:28:23.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:24.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:24.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:28:24.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:28:24.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:28:24.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:25.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:25.438Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:26.363Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:26.364Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:26.379Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:26.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:27.165Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:27.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:28.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:28.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:29.310Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:29.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:28:29.446Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:28:29.492Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:28:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:28:29.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:30.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:30.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:31.144Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154910274131011, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:28:31.144Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:31.144Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:31.144Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:28:31.145Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:28:31.145Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=507.226µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:28:31.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:31.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:32.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:32.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:33.306Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:33.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:34.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:34.415Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:28:34.415Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:28:34.415Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:28:34.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:35.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:35.541Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:36.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:36.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:36.390Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:36.582Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:37.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:37.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:38.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:38.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:39.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:39.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:40.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:40.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:41.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:41.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:42.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:42.609Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:43.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:43.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:44.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:44.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:28:44.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:28:44.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:28:44.408Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:28:44.436Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:28:44.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:28:44.516Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:28:44.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:45.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:45.611Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:46.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:46.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:46.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:46.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:47.283Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:47.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:48.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:48.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:49.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:49.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:50.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:50.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:51.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:51.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:52.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:52.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:53.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154932569600087, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:28:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:28:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:28:53.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:28:53.389Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=380.854µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:28:53.457Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:54.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:28:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:28:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:28:54.462Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:55.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:55.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:56.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:56.409Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:56.411Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:28:56.556Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:57.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:57.608Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:58.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:58.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:59.170Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:28:59.425Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:28:59.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:28:59.510Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:28:59.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:28:59.571Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:29:00.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:00.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:01.143Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154940208031146, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:29:01.143Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:01.143Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:01.143Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:29:01.143Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:29:01.144Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=511.734µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:29:01.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:01.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:02.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:02.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:03.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:03.501Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:04.249Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:04.398Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:29:04.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:29:04.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:29:04.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:05.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:05.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:06.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:06.387Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:06.387Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:06.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:07.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:07.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:08.329Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:08.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:09.249Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:09.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:10.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:10.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:11.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:11.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:12.276Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:12.443Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:13.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:13.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:14.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:14.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:29:14.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:29:14.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:29:14.405Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:29:14.475Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:29:14.509Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:29:14.563Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:29:14.626Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:15.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:15.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:16.330Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:16.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:16.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:16.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:17.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:17.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:18.307Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:18.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:19.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:19.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:20.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:20.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:21.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:21.588Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:22.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:22.611Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:23.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154962611700918, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:29:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:23.385Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:29:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:29:23.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=379.959µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:29:23.457Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:24.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:24.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:29:24.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:29:24.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:29:24.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:25.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:25.523Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:26.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:26.400Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:26.400Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:26.632Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:27.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:27.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:28.311Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:28.539Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:29.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:29.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:29:29.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:29:29.506Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:29:29.579Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:29:29.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:30.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:30.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154970217945617, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:29:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:29:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:29:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=511.363µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:29:31.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:31.446Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:32.353Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:32.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:33.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:33.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:34.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:29:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:29:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:29:34.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:35.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:35.449Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:36.286Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:36.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:36.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:36.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:37.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:37.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:38.305Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:38.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:39.254Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:39.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:40.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:40.568Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:41.224Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:41.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:42.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:42.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:43.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:43.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:44.154Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:44.419Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:29:44.487Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:29:44.487Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:29:44.487Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:29:44.507Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:29:44.540Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:29:44.562Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:29:44.718Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:45.335Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:45.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:46.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:46.353Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:46.354Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:46.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:47.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:47.568Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:48.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:48.448Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:49.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:49.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:50.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:50.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:51.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:51.615Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:52.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:52.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:53.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:53.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717154992453101913, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:29:53.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:53.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:29:53.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:29:53.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:29:53.395Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=410.64µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:29:53.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:54.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:54.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:29:54.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:29:54.404Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:29:54.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:55.224Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:55.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:56.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:56.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:56.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:29:56.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:57.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:57.447Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:58.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:58.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:29:59.167Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:29:59.422Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:29:59.460Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:29:59.518Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:29:59.520Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:29:59.593Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:00.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:00.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:01.151Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155000186261533, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:30:01.151Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:01.152Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:01.152Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:30:01.152Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:30:01.152Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=487.553µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:30:01.283Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:01.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:02.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:02.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:03.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:03.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:04.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:04.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:30:04.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:30:04.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:30:04.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:05.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:05.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:06.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:06.404Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:06.405Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:06.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:07.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:07.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:08.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:08.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:10.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:10.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:11.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:11.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:12.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:12.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:13.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:13.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:14.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:14.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:14.399Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":154443592,"TotalAlloc":91102570417328,"Sys":327718216,"Lookups":0,"Mallocs":970466433259,"Frees":970464898039,"HeapAlloc":154443592,"HeapSys":308379648,"HeapIdle":129646592,"HeapInuse":178733056,"HeapReleased":97230848,"HeapObjects":1535220,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":2972480,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191482,"GCSys":7462208,"OtherSys":1611742,"NextGC":198076904,"LastGC":1717155014394568118,"PauseTotalNs":3780678688237,"PauseNs":[14620494,20125721,17564158,1093567,19200100,148148,142647,9361581,147482,20983966,123080,20220540,139027,107792,14253457,8142468,8211070,121046,19713497,28883482,29458260,33352133,8274007,16147487,140358,11902041,129341,23147291,24090275,7044868,167150,8166036,118540,16034790,14759780,8425226,36066575,12425237,2913672,12780666,20841781,9621674,11060995,28184759,16919607,669645,6532317,4247483,18071262,8127016,10181394,99267,19735377,7254786,12806134,11047596,10449522,10212269,24927978,5615816,14442395,17430497,7739478,26099608,159625,12954075,11903991,25547180,8576984,19783720,173103,19999589,12834862,11435234,16516630,141161,977821,8146246,9626579,13067427,12076952,118836,15163880,9770172,10551223,7385917,10227049,156580,19545105,24592456,13193281,10078836,17232653,14213313,1330612,130886,29971861,11238660,11946113,6148996,7311146,144182,8977609,12308754,2797933,229783,94123,30129050,26259173,16199173,4132476,16159389,2831143,14782740,11715993,9180798,11926310,12401704,11666905,5199175,7880463,14391681,229870,24808994,4191806,17253022,11420023,21432340,143573,11000597,12225029,8504860,11795765,121696,31144377,24768906,19738111,12769825,18933432,16246548,7142941,16184929,23603815,3438372,32659279,161784,127527,11809263,14567394,13151692,1920078,25018241,8100856,3275556,190748,16620138,8181453,12260767,12050020,14625166,18787488,2471620,152876,147839,113263,139555,130987,285805,165548,297944,136628,130101,287383,111194,140882,158442,120694,134958,300399,133124,137476,129534,141082,138368,129802,127205,130613,137649,135240,132513,134278,131714,388486,131006,139688,122338,137649,122448,137497,162304,119459,118570,127531,189973,96916,8139712,20140019,3618786,12927650,740183,11032073,9530143,158678,11954336,156580,30910506,19036217,3503086,22311168,8110935,13366971,24687042,28064841,15930698,331409,6611031,19887408,12172610,16925092,176728,20201450,13441465,7983699,16477017,22728506,4111546,17297039,25326186,12158743,13948344,19467425,12134039,8163847,16144210,10302689,4932134,2278777,4357358,15721415,16155093,8196323,170659,8106844,15796773,4154815,16373727],"PauseEnd":[1717154915405207585,1717154915931922054,1717154916507376339,1717154916976086804,1717154917541962305,1717154918267162010,1717154918813973631,1717154919362136580,1717154919899441276,1717154920480211949,1717154920943272018,1717154921545918739,1717154922008604280,1717154922624159194,1717154923310614989,1717154923833104478,1717154924431960206,1717154924964017679,1717154925562415454,1717154926061319780,1717154926729765746,1717154927324374416,1717154927972304664,1717154928643305396,1717154929146681578,1717154929733640160,1717154930271008705,1717154930793988481,1717154931412258623,1717154931907789335,1717154932449334949,1717154933104925502,1717154933668218579,1717154934166522115,1717154934729184997,1717154935322963614,1717154935890239916,1717154936452744983,1717154936963024923,1717154937571001255,1717154938343099146,1717154938885783592,1717154939460959048,1717154940008090890,1717154940613392501,1717154941211509584,1717154942185174756,1717154943127254680,1717154943794465156,1717154944397926080,1717154944862641573,1717154945418867131,1717154945941261234,1717154946512667194,1717154946995125604,1717154947587035904,1717154948290508585,1717154948812885645,1717154949419390223,1717154949915426730,1717154950548581735,1717154951050996367,1717154951669607388,1717154952204395022,1717154952746237231,1717154953487799014,1717154954006659937,1717154954622312959,1717154955105370384,1717154955704849433,1717154956261392353,1717154956844695141,1717154957440365404,1717154958150889943,1717154958762068076,1717154959310358445,1717154959807300287,1717154960386808465,1717154960901060099,1717154961463376424,1717154961973464215,1717154962499004710,1717154963165062161,1717154963760892699,1717154964315149577,1717154964811412162,1717154965348843208,1717154965884595704,1717154966551644006,1717154967146506356,1717154967713808092,1717154968398722145,1717154968905694366,1717154969505217635,1717154969992073700,1717154970587896818,1717154971164975090,1717154971718162241,1717154972269128698,1717154972847027044,1717154973535039404,1717154974073708095,1717154974648731356,1717154975127145050,1717154975648556975,1717154976165435994,1717154976816712359,1717154977488498455,1717154978255314306,1717154978821689391,1717154979391816826,1717154979950732202,1717154980513623265,1717154981052211877,1717154981679577085,1717154982167665705,1717154982745138194,1717154983432876248,1717154983958708137,1717154984717381537,1717154985240026826,1717154985799477669,1717154986382623714,1717154986921845771,1717154987460866328,1717154988177152616,1717154988760351942,1717154989340046816,1717154989918911607,1717154990464836785,1717154990995065441,1717154991638459609,1717154992181715735,1717154992686745170,1717154993445750210,1717154993972490734,1717154994611319398,1717154995101095712,1717154995666999998,1717154996149750630,1717154996704604724,1717154997226732713,1717154997846913001,1717154998471729175,1717154998943175098,1717154999486673916,1717154999986192650,1717155000585806384,1717155001122965893,1717155001790906815,1717155002426312617,1717155003130965817,1717155003776746303,1717155004335002837,1717155004845936263,1717155005410498645,1717155005945149251,1717155006568678548,1717155007070719858,1717155007686000535,1717155008437935572,1717155008925439248,1717155009103076833,1717155009234144926,1717155009378527698,1717155009517174726,1717155009646520136,1717155009777703405,1717155009906116295,1717155010035987288,1717155010165262446,1717155010287997200,1717155010414756400,1717155010544907467,1717155010674113430,1717155010796079006,1717155010918988144,1717155011050298591,1717155011175673679,1717155011306964607,1717155011438155234,1717155011564238360,1717155011693012745,1717155011823836705,1717155011945121937,1717155012083905326,1717155012213319273,1717155012347122573,1717155012475853389,1717155012598768231,1717155012726879131,1717155012866994391,1717155012997339587,1717155013113366452,1717155013235905876,1717155013380772802,1717155013514138639,1717155013639997195,1717155013768378598,1717155013897106584,1717155014021771536,1717155014141124327,1717155014268681804,1717155014394568118,1717154885264429208,1717154885822262128,1717154886417471507,1717154886952326361,1717154887523738453,1717154888260188114,1717154888853175304,1717154889396443239,1717154889954461928,1717154890580904143,1717154891065322867,1717154891702493900,1717154892216918482,1717154892808453022,1717154893460557399,1717154893995764539,1717154894631750420,1717154895154406361,1717154895729329522,1717154896304317332,1717154896798564778,1717154897437496536,1717154898153944947,1717154898744832020,1717154899331764981,1717154899866384599,1717154900404422717,1717154900930445788,1717154901540361935,1717154902023242128,1717154902628927046,1717154903372159444,1717154903982077334,1717154904632935560,1717154905134614356,1717154905707759833,1717154906293521607,1717154906855630407,1717154907418825240,1717154908099029358,1717154908690381950,1717154909217609780,1717154909768905887,1717154910299683052,1717154910809815486,1717154911416447237,1717154911959120990,1717154912522818764,1717154913212541968,1717154913793545926,1717154914405142186,1717154914879226420],"NumGC":1077964,"NumForcedGC":0,"GCCPUFraction":0.03636803972203369,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437599267,"Frees":437592720},{"Size":16,"Mallocs":6772587535,"Frees":6772233373},{"Size":24,"Mallocs":19152718609,"Frees":19152587257},{"Size":32,"Mallocs":10366469857,"Frees":10366394543},{"Size":48,"Mallocs":1990750936,"Frees":1990705051},{"Size":64,"Mallocs":240651035,"Frees":240620460},{"Size":80,"Mallocs":238900447,"Frees":238885205},{"Size":96,"Mallocs":915851101250,"Frees":915850323052},{"Size":112,"Mallocs":383469110,"Frees":383465142},{"Size":128,"Mallocs":9678165067,"Frees":9678150492},{"Size":144,"Mallocs":190302651,"Frees":190298265},{"Size":160,"Mallocs":71586201,"Frees":71584213},{"Size":176,"Mallocs":5497865,"Frees":5497065},{"Size":192,"Mallocs":32380038,"Frees":32379599},{"Size":208,"Mallocs":68268047,"Frees":68264070},{"Size":224,"Mallocs":3199705,"Frees":3199179},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52396156,"Frees":52368631},{"Size":288,"Mallocs":98617387,"Frees":98598891},{"Size":320,"Mallocs":49976181,"Frees":49972726},{"Size":352,"Mallocs":15142503,"Frees":15141972},{"Size":384,"Mallocs":30443271,"Frees":30443103},{"Size":416,"Mallocs":5927915,"Frees":5925437},{"Size":448,"Mallocs":5902189,"Frees":5897882},{"Size":480,"Mallocs":642394,"Frees":642042},{"Size":512,"Mallocs":39609801,"Frees":39609600},{"Size":576,"Mallocs":21104380,"Frees":21101951},{"Size":640,"Mallocs":1232981,"Frees":1232736},{"Size":704,"Mallocs":18219450,"Frees":18218736},{"Size":768,"Mallocs":6979747,"Frees":6979424},{"Size":896,"Mallocs":7320739,"Frees":7320336},{"Size":1024,"Mallocs":52560762,"Frees":52560586},{"Size":1152,"Mallocs":24976774,"Frees":24976705},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16254795,"Frees":16254766},{"Size":1536,"Mallocs":1800332,"Frees":1800328},{"Size":1792,"Mallocs":12381749,"Frees":12379669},{"Size":2048,"Mallocs":710041,"Frees":709945},{"Size":2304,"Mallocs":22949007,"Frees":22948487},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2037456,"Frees":2037439},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8464268,"Frees":8464264},{"Size":4096,"Mallocs":3537713,"Frees":3537629},{"Size":4864,"Mallocs":8029154,"Frees":8029119},{"Size":5376,"Mallocs":1749558,"Frees":1748014},{"Size":6144,"Mallocs":1312057,"Frees":1312047},{"Size":6528,"Mallocs":6063959,"Frees":6063953},{"Size":6784,"Mallocs":1759296,"Frees":1759226},{"Size":6912,"Mallocs":249867,"Frees":249867},{"Size":8192,"Mallocs":226091,"Frees":225989},{"Size":9472,"Mallocs":1839222,"Frees":1839200},{"Size":9728,"Mallocs":106689,"Frees":106689},{"Size":10240,"Mallocs":1188600,"Frees":1188600},{"Size":10880,"Mallocs":33562,"Frees":33562},{"Size":12288,"Mallocs":273029,"Frees":273025},{"Size":13568,"Mallocs":7822944,"Frees":7822802},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1389293,"Frees":1389264},{"Size":18432,"Mallocs":1190967,"Frees":1190961}]} 2024-05-31T11:30:14.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:30:14.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:30:14.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:30:14.400Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:30:14.416Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:30:14.431Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:30:14.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:30:15.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:15.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:16.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:16.330Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:16.330Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:16.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:17.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:17.382Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:18.094Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:18.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:19.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:19.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:20.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:20.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:21.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:21.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:22.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:22.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:23.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:23.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155023367450371, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:30:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:23.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:23.385Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:30:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:30:23.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=343.824µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:30:24.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:24.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:30:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:30:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:30:25.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:25.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:26.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:26.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:26.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:26.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:27.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:27.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:28.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:28.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:29.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:29.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:30:29.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:30:29.427Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:30:29.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:30:30.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:30.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:31.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:31.137Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155031083399601, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:30:31.137Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:31.137Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:31.137Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:30:31.137Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:30:31.138Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=485.984µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:30:31.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:32.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:32.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:33.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:33.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:34.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:34.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:30:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:30:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:30:35.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:35.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:36.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:36.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:37.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:37.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:38.092Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:38.379Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:39.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:39.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:40.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:40.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:41.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:41.384Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:42.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:42.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:43.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:43.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:44.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:44.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:30:44.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:30:44.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:30:44.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:30:44.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:30:44.429Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:30:44.506Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:30:45.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:45.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:46.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:46.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:46.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:46.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:47.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:47.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:48.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:48.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:49.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:49.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:50.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:50.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:51.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:51.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:52.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:52.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:53.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:53.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:53.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155053358191606, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:30:53.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:53.385Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:30:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:30:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:30:53.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=326.718µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:30:54.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:54.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:30:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:30:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:30:55.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:55.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:56.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:56.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:56.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:30:56.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:57.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:57.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:58.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:58.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:59.067Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:30:59.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:30:59.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:30:59.413Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:30:59.429Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:30:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:31:00.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:00.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:01.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155061069047176, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:31:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:31:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:31:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=459.47µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:31:01.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:02.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:02.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:03.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:03.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:04.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:04.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:31:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:31:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:31:05.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:05.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:06.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:06.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:06.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:06.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:07.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:07.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:08.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:08.383Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:09.128Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:09.396Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:10.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:10.413Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:11.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:11.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:12.119Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:12.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:13.111Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:13.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:14.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:14.410Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:31:14.440Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:31:14.440Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:31:14.440Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:31:14.470Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:31:14.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:31:14.546Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:31:14.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:15.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:15.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:16.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:16.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:16.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:16.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:17.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:17.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:18.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:18.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:19.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:19.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:20.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:20.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:21.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:21.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:22.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:22.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:23.342Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155082602543652, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:31:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:31:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:31:23.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=372.085µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:31:23.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:24.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:24.419Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:31:24.419Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:31:24.419Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:31:24.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:25.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:25.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:26.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:26.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:26.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:26.672Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:27.330Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:27.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:28.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:28.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:29.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:29.399Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:31:29.456Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:31:29.502Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:31:29.532Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:31:29.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:30.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:30.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:31.139Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155090204875305, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:31:31.139Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:31.139Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:31.139Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:31:31.139Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:31:31.140Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=494.268µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:31:31.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:31.462Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:32.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:32.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:33.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:33.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:33.829Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Start checkpointing for replication 2024-05-31T11:31:34.108Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:31:34.132Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Checkpointing for vb_list=[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511], time_to_wait=10m0s, interval_btwn_vb=1.171 sec 2024-05-31T11:31:34.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:34.409Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:31:34.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:31:34.411Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:31:34.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:35.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:35.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:36.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:36.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:36.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:37.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:37.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:37.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:38.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:38.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:39.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:39.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:40.283Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:40.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:41.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:41.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:42.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:42.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:43.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:43.523Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:44.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:44.426Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:31:44.426Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:31:44.426Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:31:44.426Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:31:44.494Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:31:44.516Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:31:44.566Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:31:44.566Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:31:44.567Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:31:44.568Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:31:44.569Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:31:44.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:45.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:45.615Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:46.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:46.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:46.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:46.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:47.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:47.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:48.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:48.623Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:49.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:49.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:50.329Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:50.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:51.158Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:51.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:52.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:52.604Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:53.402Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:53.419Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155112605258386, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:31:53.419Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:53.419Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:31:53.419Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:31:53.419Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:31:53.419Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=384.747µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:31:53.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:54.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:54.431Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:31:54.431Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:31:54.431Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:31:54.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:55.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:55.696Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:56.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:56.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:56.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:31:56.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:57.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:57.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:58.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:58.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:59.258Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:31:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:31:59.460Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:31:59.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:31:59.484Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:31:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:31:59.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:31:59.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:31:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:31:59.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:32:00.311Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:00.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:01.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155120312368000, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:32:01.181Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:01.181Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:01.182Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:32:01.182Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:32:01.182Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=547.053µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:32:01.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:01.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:02.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:02.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:03.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:03.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:04.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:04.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:32:04.409Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:32:04.409Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:32:04.582Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:05.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:05.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:06.298Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:06.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:06.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:06.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:07.345Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:07.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:08.258Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:08.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:09.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:09.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:10.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:10.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:11.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:11.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:12.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:12.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:13.345Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:13.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:14.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:14.428Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:32:14.429Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:32:14.430Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:32:14.430Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:32:14.444Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":189853544,"TotalAlloc":91151584649064,"Sys":327718216,"Lookups":0,"Mallocs":970983360401,"Frees":970981430427,"HeapAlloc":189853544,"HeapSys":308379648,"HeapIdle":95887360,"HeapInuse":212492288,"HeapReleased":63160320,"HeapObjects":1929974,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3605600,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3191850,"GCSys":7462208,"OtherSys":1611374,"NextGC":202542560,"LastGC":1717155133917896723,"PauseTotalNs":3781919125974,"PauseNs":[122975,186902,150123,126795,124792,105123,288756,135668,122058,124496,136735,124814,122060,123816,114838,295642,132847,121027,118416,129644,127012,138977,121905,138926,119933,145810,128429,131028,132085,127200,136308,125954,125766,132275,111392,135499,129154,127194,120882,116854,136315,120594,132009,122343,115532,126950,119955,131734,137386,133290,113169,112739,111991,129315,136068,129110,129144,133288,159409,127022,116724,134052,112620,296958,134646,133007,111776,122012,129715,144503,133453,296888,140214,134937,138489,126879,125006,110350,297330,130971,127569,111833,125455,132988,142571,122700,136810,114641,161297,111614,298463,130314,130824,111743,3438174,133243,124944,132501,126060,134814,285698,139855,131122,150269,133039,120958,199771,123063,124895,131576,127345,128796,129987,124237,320822,282995,150815,132881,126640,142955,139361,118236,136176,129364,137121,135602,152987,145114,131481,138348,4126737,170194,326532,130566,5091580,139276,127083,135036,8973942,155044,6194149,147393,123688,111239,18350988,12177425,12139341,162413,7329918,13451502,1252894,5109889,3592255,12152003,2496471,8029747,327982,6125563,308395,155619,3941692,11872360,17093561,19859003,13810320,3022722,13017960,12077256,12919443,22251328,8296774,15109186,8116430,4875264,5965959,12147218,9373691,8545383,7315981,8899980,337071,18810378,157940,11845507,14939607,22939602,18537143,17079463,21713125,155628,16085072,8839493,18374743,20245278,12615217,13642159,7013545,644660,9159744,139888,13895880,135400,20211363,12599364,9449184,25960997,4108886,99103,17129355,22851917,9356775,150442,33008988,7693209,8233537,28866536,103121,16387484,30471260,9490221,14901974,2773039,12227005,8186814,16030817,141997,23389283,8461554,25772519,19337027,345646,96168,12119870,8732025,12157476,118993,581580,21687831,3935471,18823784,12640391,11444001,7227573,20920143,13836708,13617377,17827322,11382251,331356,129326,292294,130135,121285,128026,122518,156636],"PauseEnd":[1717155053438314659,1717155053558961533,1717155053681539133,1717155053815103923,1717155053947122593,1717155054082583222,1717155054206713997,1717155054328769243,1717155054454845398,1717155054584376348,1717155054712857507,1717155054833279925,1717155054946020458,1717155055083680490,1717155055212337598,1717155055328511196,1717155055447146231,1717155055566899504,1717155055691203927,1717155055811072170,1717155055933290360,1717155056083045470,1717155056218745735,1717155056350382528,1717155056476277923,1717155056601990851,1717155056730351089,1717155056859343833,1717155056988251919,1717155057107784477,1717155057232033367,1717155057371744388,1717155057497583081,1717155057620944122,1717155057746502633,1717155057876599917,1717155058003231632,1717155058121019620,1717155058244803258,1717155058381142817,1717155058513976408,1717155058633280047,1717155058755236141,1717155058883130915,1717155059011383766,1717155059127133330,1717155059249365491,1717155059384047210,1717155059515945385,1717155059643893503,1717155059770660975,1717155059890326870,1717155060009981661,1717155060131716912,1717155060258697018,1717155060390602977,1717155060524002380,1717155060643559008,1717155060765538181,1717155060892263810,1717155061019515042,1717155061133605070,1717155061255111330,1717155061388778742,1717155061515812832,1717155061632915533,1717155061754150448,1717155061874428373,1717155061995537106,1717155062115099179,1717155062236825924,1717155062372500140,1717155062502557648,1717155062623913750,1717155062739580904,1717155062869671604,1717155062997380868,1717155063122414796,1717155063252893478,1717155063384204194,1717155063510828763,1717155063620822753,1717155063743986671,1717155063873460085,1717155064015694300,1717155064161048782,1717155064309039940,1717155064458626063,1717155064594974134,1717155064731217202,1717155064870635266,1717155065012392331,1717155065146662999,1717155065278830858,1717155065421312946,1717155065560677626,1717155065703642202,1717155065839198225,1717155065969979331,1717155066115082728,1717155066260938727,1717155066409270704,1717155066544843660,1717155066686876767,1717155066842862919,1717155066978164348,1717155067131919283,1717155067273384625,1717155067412314400,1717155067549633919,1717155067692988036,1717155067883075483,1717155068039913084,1717155068187992136,1717155068335430582,1717155068480890934,1717155068620009900,1717155068758114399,1717155068889092594,1717155069057151212,1717155069252109536,1717155069448655995,1717155069654060557,1717155069825910190,1717155070010872879,1717155070230465489,1717155070455794803,1717155070636531895,1717155070831460043,1717155070998311472,1717155071234818631,1717155071451823946,1717155071644115306,1717155071856791016,1717155072061663110,1717155072254990733,1717155072476567959,1717155072668525544,1717155072887059256,1717155073134669678,1717155073347367147,1717155073548445829,1717155073737163549,1717155073933963612,1717155074440734149,1717155074966726374,1717155075492949164,1717155075930869831,1717155076513654560,1717155077035106563,1717155077675117050,1717155078360982101,1717155078843102047,1717155079400833326,1717155079847851382,1717155080421148527,1717155080894223614,1717155081443424285,1717155081952338786,1717155082538764283,1717155083253806336,1717155083800645399,1717155084387404625,1717155084872334299,1717155085399124197,1717155085888459957,1717155086550130572,1717155087199979638,1717155087793788800,1717155088483675452,1717155088969692376,1717155089566515961,1717155090040121911,1717155090614154504,1717155091091217851,1717155091689423261,1717155092147900609,1717155092708943468,1717155093361916734,1717155093879927649,1717155094550007118,1717155095070480886,1717155095632254423,1717155096076220854,1717155096670920147,1717155097376166215,1717155098054932791,1717155098684855617,1717155099221821762,1717155099792117520,1717155100390290900,1717155100918990898,1717155101574237898,1717155102059744743,1717155102657319302,1717155103379654491,1717155103909041433,1717155104515144453,1717155105027965551,1717155105613616642,1717155106119050226,1717155106737461100,1717155107300771150,1717155107929988214,1717155108621368568,1717155109150435767,1717155109716828337,1717155110256649028,1717155110843921489,1717155111435229286,1717155111941125482,1717155112532429268,1717155113328077026,1717155113904433219,1717155114499110229,1717155114982273645,1717155115609823249,1717155116163627223,1717155116754837894,1717155117291373346,1717155117882903276,1717155118619877682,1717155119157429287,1717155119728037672,1717155120234287411,1717155120764443819,1717155121369687170,1717155121931402462,1717155122541978640,1717155123230410522,1717155123937936354,1717155124549231201,1717155125075503191,1717155125654957547,1717155126209112743,1717155126744369793,1717155127247337019,1717155127826706921,1717155128503523358,1717155129077902351,1717155129657455508,1717155130208268843,1717155130847938035,1717155131472135437,1717155131993181392,1717155132591619426,1717155133318041711,1717155133917896723,1717155052377719561,1717155052518211242,1717155052645318091,1717155052792380144,1717155052924761581,1717155053054381946,1717155053183057766,1717155053318965970],"NumGC":1078520,"NumForcedGC":0,"GCCPUFraction":0.0363771789132723,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437718611,"Frees":437711882},{"Size":16,"Mallocs":6774590824,"Frees":6774227897},{"Size":24,"Mallocs":19157936220,"Frees":19157779354},{"Size":32,"Mallocs":10369268266,"Frees":10369178971},{"Size":48,"Mallocs":1991338743,"Frees":1991289110},{"Size":64,"Mallocs":240697115,"Frees":240665433},{"Size":80,"Mallocs":238945082,"Frees":238929835},{"Size":96,"Mallocs":916352768618,"Frees":916351652734},{"Size":112,"Mallocs":383560300,"Frees":383556334},{"Size":128,"Mallocs":9680798191,"Frees":9680781293},{"Size":144,"Mallocs":190354480,"Frees":190350088},{"Size":160,"Mallocs":71596654,"Frees":71594665},{"Size":176,"Mallocs":5498453,"Frees":5497647},{"Size":192,"Mallocs":32389321,"Frees":32388865},{"Size":208,"Mallocs":68284434,"Frees":68280415},{"Size":224,"Mallocs":3200059,"Frees":3199533},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52405431,"Frees":52377875},{"Size":288,"Mallocs":98638448,"Frees":98618889},{"Size":320,"Mallocs":49982755,"Frees":49979287},{"Size":352,"Mallocs":15145558,"Frees":15144999},{"Size":384,"Mallocs":30452269,"Frees":30452085},{"Size":416,"Mallocs":5929285,"Frees":5926803},{"Size":448,"Mallocs":5902216,"Frees":5897909},{"Size":480,"Mallocs":642394,"Frees":642042},{"Size":512,"Mallocs":39620895,"Frees":39620642},{"Size":576,"Mallocs":21109324,"Frees":21106888},{"Size":640,"Mallocs":1233242,"Frees":1232997},{"Size":704,"Mallocs":18223867,"Frees":18223152},{"Size":768,"Mallocs":6981925,"Frees":6981591},{"Size":896,"Mallocs":7322842,"Frees":7322436},{"Size":1024,"Mallocs":52574386,"Frees":52574174},{"Size":1152,"Mallocs":24982976,"Frees":24982895},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16259200,"Frees":16259171},{"Size":1536,"Mallocs":1800830,"Frees":1800825},{"Size":1792,"Mallocs":12386083,"Frees":12383986},{"Size":2048,"Mallocs":710216,"Frees":710117},{"Size":2304,"Mallocs":22954634,"Frees":22954106},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2037852,"Frees":2037835},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8467216,"Frees":8467198},{"Size":4096,"Mallocs":3538481,"Frees":3538393},{"Size":4864,"Mallocs":8030424,"Frees":8030387},{"Size":5376,"Mallocs":1749626,"Frees":1748082},{"Size":6144,"Mallocs":1312082,"Frees":1312072},{"Size":6528,"Mallocs":6066135,"Frees":6066123},{"Size":6784,"Mallocs":1760068,"Frees":1759995},{"Size":6912,"Mallocs":249927,"Frees":249927},{"Size":8192,"Mallocs":226141,"Frees":226039},{"Size":9472,"Mallocs":1839854,"Frees":1839830},{"Size":9728,"Mallocs":106737,"Frees":106737},{"Size":10240,"Mallocs":1188839,"Frees":1188838},{"Size":10880,"Mallocs":33574,"Frees":33574},{"Size":12288,"Mallocs":273097,"Frees":273093},{"Size":13568,"Mallocs":7825892,"Frees":7825726},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1389686,"Frees":1389655},{"Size":18432,"Mallocs":1191405,"Frees":1191395}]} 2024-05-31T11:32:14.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:32:14.507Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:32:14.542Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:32:14.543Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:32:14.543Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:32:14.546Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:32:14.562Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:32:14.584Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:15.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:15.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:16.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:16.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:16.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:16.444Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:17.315Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:17.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:18.286Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:18.613Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:19.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:19.444Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:20.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:20.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:21.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:21.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:22.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:22.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:23.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:23.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155142562706060, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:32:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:32:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:32:23.390Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=380.398µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:32:23.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:24.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:24.400Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:32:24.401Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:32:24.401Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:32:24.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:25.167Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:25.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:26.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:26.389Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:26.390Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:26.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:27.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:27.600Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:28.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:28.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:29.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:29.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:32:29.427Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:32:29.480Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:32:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:32:29.503Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:32:29.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:32:29.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:32:29.511Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:32:29.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:30.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:30.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:31.142Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155150198083707, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:32:31.143Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:31.143Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:31.144Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:32:31.145Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:32:31.146Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=672.884µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:32:31.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:31.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:32.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:32.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:33.392Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:33.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:34.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:34.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:32:34.409Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:32:34.409Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:32:34.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:35.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:35.434Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:36.353Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:36.353Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:36.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:36.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:37.331Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:37.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:38.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:38.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:39.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:39.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:40.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:40.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:41.355Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:41.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:42.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:42.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:43.244Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:43.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:44.341Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:32:44.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:32:44.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:32:44.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:32:44.440Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:32:44.480Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:32:44.507Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:32:44.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:45.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:45.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:46.145Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:46.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:46.358Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:46.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:47.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:47.447Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:48.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:48.578Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:49.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:49.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:50.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:50.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:51.329Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:51.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:52.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:52.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:53.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155172505859698, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:32:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:32:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:32:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:32:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=357.748µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:32:53.599Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:54.075Z INFO GOXDCR.P2PManager: Successfully sent request type PeriodicPush to nodes with opaque: map[10.0.9.103:8091:903806976] 2024-05-31T11:32:54.203Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Start checkpointing for replication 2024-05-31T11:32:54.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:32:54.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:32:54.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:32:54.429Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:54.601Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:32:54.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:54.621Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Checkpointing for vb_list=[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511], time_to_wait=10m0s, interval_btwn_vb=1.171 sec 2024-05-31T11:32:54.708Z INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type PeriodicPush from 10.0.9.103:8091 2024-05-31T11:32:54.709Z INFO GOXDCR.P2PManager: Received peer-to-peer push requests from 10.0.9.103:8091 (opaque 1229651968) 2024-05-31T11:32:55.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:55.626Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:56.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:56.335Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:56.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:32:56.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:57.310Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:57.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:58.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:58.671Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:32:59.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:32:59.402Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:32:59.457Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:32:59.504Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:32:59.504Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:32:59.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:00.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:00.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:01.137Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155180190843038, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:33:01.137Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:01.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:01.138Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:33:01.138Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:33:01.138Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=527.988µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:33:01.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:01.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:02.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:02.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:03.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:03.593Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:04.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:04.398Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:33:04.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:33:04.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:33:04.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:05.265Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:05.523Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:06.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:06.352Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:06.371Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:06.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:07.258Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:07.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:08.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:08.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:09.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:10.295Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:10.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:11.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:11.553Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:12.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:12.637Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:13.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:13.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:14.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:14.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:33:14.416Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:33:14.416Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:33:14.416Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:33:14.469Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:33:14.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:33:14.525Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:33:14.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:15.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:15.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:16.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:16.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:16.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:16.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:17.375Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:17.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:18.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:18.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:19.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:19.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:20.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:20.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:21.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:21.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:22.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:22.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:23.253Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:23.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155202550399789, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:33:23.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:23.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:23.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:33:23.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:33:23.396Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=354.696µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:33:23.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:24.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:24.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:33:24.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:33:24.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:33:24.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:25.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:25.588Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:26.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:26.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:26.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:26.633Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:27.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:27.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:28.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:28.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:29.318Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:29.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:33:29.432Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:33:29.477Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:33:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:33:29.497Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:33:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:33:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:33:29.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:29.521Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:33:30.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:30.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155210195123299, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:33:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:33:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:33:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=498.806µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:33:31.210Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:31.595Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:32.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:32.455Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:33.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:33.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:34.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:34.424Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:33:34.425Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:33:34.426Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:33:34.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:35.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:35.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:36.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:36.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:36.369Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:37.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:37.404Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:38.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:38.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:39.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:39.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:40.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:40.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:41.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:41.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:42.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:42.578Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:43.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:43.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:44.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:33:44.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:33:44.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:33:44.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:33:44.438Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:33:44.454Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:33:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:33:44.520Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:45.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:45.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:46.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:46.331Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:46.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:46.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:47.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:47.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:48.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:48.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:49.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:49.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:50.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:50.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:51.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:51.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:52.196Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:52.538Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:53.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155232538745261, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:33:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:33:53.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:33:53.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:33:53.389Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=355.929µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:33:53.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:54.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:33:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:33:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:33:54.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:55.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:55.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:56.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:56.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:56.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:33:56.578Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:57.300Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:57.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:58.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:58.578Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:59.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:33:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:33:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:33:59.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:33:59.510Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:33:59.589Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:34:00.300Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:00.445Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:01.141Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155240300562276, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:34:01.142Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:01.142Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:01.143Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:34:01.144Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:34:01.161Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=16.681775ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:34:01.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:01.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:02.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:02.575Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:03.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:03.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:04.267Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:04.409Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:34:04.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:34:04.410Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:34:04.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:05.160Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:05.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:06.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:06.339Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:06.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:06.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:07.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:07.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:08.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:08.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:09.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:09.434Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:10.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:10.463Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:11.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:11.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:12.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:12.623Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:13.294Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:13.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:14.143Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:14.399Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":204134456,"TotalAlloc":91172621267720,"Sys":327718216,"Lookups":0,"Mallocs":971208545214,"Frees":971206519825,"HeapAlloc":204134456,"HeapSys":308346880,"HeapIdle":82698240,"HeapInuse":225648640,"HeapReleased":59670528,"HeapObjects":2025389,"StackInuse":1900544,"StackSys":1900544,"MSpanInuse":3780640,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193682,"GCSys":7462208,"OtherSys":1609542,"NextGC":211338352,"LastGC":1717155253828113004,"PauseTotalNs":3784152943467,"PauseNs":[3494630,19445811,651420,14776983,9810327,11329660,21035173,28197727,3898391,10538662,153862,27012751,23807890,15089239,9827548,120504,28371193,11916683,14349544,8632683,19911845,15487342,12081380,429649,25579893,12223110,2221505,12105104,7706833,131360,26439425,141718,84416,611821,20256782,18213927,13749991,102882,9028485,9534955,8149740,8163935,18970681,21729301,20645186,14949548,5392728,12114288,8156268,4104833,21501429,2953001,11389147,8344696,20666819,9211932,96792,12117997,6625057,15007447,179932,20725177,18080614,2199828,25858523,1845634,17307824,11073855,4281712,17172213,24541816,6378758,4362248,144426,4065180,14577934,9120068,9278052,145049,9583707,142595,2534337,8612322,24062045,16631592,148690,20594506,14838032,4659975,19135915,13263938,21455025,27101005,4097333,23287981,8046541,9712386,15906539,142619,144053,13874839,9354741,12037996,11906426,13830554,142476,24666023,16182723,23085427,160995,11961948,7847315,3404493,2887042,22675624,147523,17937764,2286776,21758776,22125027,140390,23500834,19508522,187711,12113771,6598252,5207650,133655,15811557,15583838,20663638,17646362,10026967,143324,20190734,12180414,4019870,16136190,3746554,12128908,17886236,16453766,11603467,141706,11797056,145990,165523,8146138,14661968,21384977,120498,2633076,146911,2266751,7149313,8200681,94323,144945,13401186,21344663,14970486,12755538,180615,8156249,14141575,16932752,21653801,13047927,15369641,25423389,18193540,154465,8549076,1379182,10553411,20774415,15908605,143795,306249,11542827,4177258,25346070,18071577,16465002,4689878,21086623,14997901,16873111,13667971,8182425,4131989,16605893,16147023,19138673,146714,13642159,7013545,644660,9159744,139888,13895880,135400,20211363,12599364,9449184,25960997,4108886,99103,17129355,22851917,9356775,150442,33008988,7693209,8233537,28866536,103121,16387484,30471260,9490221,14901974,2773039,12227005,8186814,16030817,141997,23389283,8461554,25772519,19337027,345646,96168,12119870,8732025,12157476,118993,581580,21687831,3935471,18823784,12640391,11444001,7227573,20920143,13836708,13617377,17827322,11382251,139416,14973295,8056277,146848,134207,3677021,8212080,22545992],"PauseEnd":[1717155139161126671,1717155139731600593,1717155140254843202,1717155140812110351,1717155141382673259,1717155141917131430,1717155142472351325,1717155143042805059,1717155143737658750,1717155144364471994,1717155144865142964,1717155145378091806,1717155145895132386,1717155146468679906,1717155146975737594,1717155147519128865,1717155148178540770,1717155148789946359,1717155149357578518,1717155149896748702,1717155150427786429,1717155150933396788,1717155151548186898,1717155152055405439,1717155152651136236,1717155153358911046,1717155153910326938,1717155154580308060,1717155155127144434,1717155155674248095,1717155156242766336,1717155156767163100,1717155157284625526,1717155157859148098,1717155158570348183,1717155159095249588,1717155159683948908,1717155160225889378,1717155160776037438,1717155161350532359,1717155161897683633,1717155162464539761,1717155163027960295,1717155163719421826,1717155164278201499,1717155164909149866,1717155165439738422,1717155165989897689,1717155166609290821,1717155167163977032,1717155167775171136,1717155168474782238,1717155168973205589,1717155169596995563,1717155170139110115,1717155170661337346,1717155171238797789,1717155171822002880,1717155172370999759,1717155172940539667,1717155173632509434,1717155174215468563,1717155174983264193,1717155175606208787,1717155176156190892,1717155176719176570,1717155177306573194,1717155177929164953,1717155178749120967,1717155179367639006,1717155179998420424,1717155180611054138,1717155181205587427,1717155181740547608,1717155182336391120,1717155182921591273,1717155183660812588,1717155184207548163,1717155184781580300,1717155185389810433,1717155185909690721,1717155186529615855,1717155187087350428,1717155187746456882,1717155188462545608,1717155188979210638,1717155189600217366,1717155190170382499,1717155190756834868,1717155191413740082,1717155191953294517,1717155192577093623,1717155193213132779,1717155193868155367,1717155194451743944,1717155195009903234,1717155195601463965,1717155196146741807,1717155196687187865,1717155197221156718,1717155197818809658,1717155198517525671,1717155199022397200,1717155199660570581,1717155200195602545,1717155200778100357,1717155201352599387,1717155201886486229,1717155202478459122,1717155203082073750,1717155203769371612,1717155204362982711,1717155204889380736,1717155205434704767,1717155206024799880,1717155206716527086,1717155207409308709,1717155207963525611,1717155208719480120,1717155209314318346,1717155209841477699,1717155210411688982,1717155210965756155,1717155211612311108,1717155212170170020,1717155212765409885,1717155213446679557,1717155213964454459,1717155214625442753,1717155215147655807,1717155215726103282,1717155216389580378,1717155217196800520,1717155217848716978,1717155218546183661,1717155219059698371,1717155219674422820,1717155220204521060,1717155220720924647,1717155221336888307,1717155221912390827,1717155222511671515,1717155223038578988,1717155223718837664,1717155224272767181,1717155224841251409,1717155225465720950,1717155226027592820,1717155226650490549,1717155227200743863,1717155227779003941,1717155228493152474,1717155229018823406,1717155229631626089,1717155230141910191,1717155230704424672,1717155231258882061,1717155231813101061,1717155232403555393,1717155232991439921,1717155233720305034,1717155234300925090,1717155234859528528,1717155235453417854,1717155236015805036,1717155236671170984,1717155237209988000,1717155237786543990,1717155238522877606,1717155239048977017,1717155239672413452,1717155240174821924,1717155240721570595,1717155241270535207,1717155241896000906,1717155242529150541,1717155243094262278,1717155243779977799,1717155244357310225,1717155244954813657,1717155245532562432,1717155246079172480,1717155246707544933,1717155247297220945,1717155247838629353,1717155248566585725,1717155249097468236,1717155249660205509,1717155250178938337,1717155250748638819,1717155251387412745,1717155251946658857,1717155252590022425,1717155253241448873,1717155253828113004,1717155103379654491,1717155103909041433,1717155104515144453,1717155105027965551,1717155105613616642,1717155106119050226,1717155106737461100,1717155107300771150,1717155107929988214,1717155108621368568,1717155109150435767,1717155109716828337,1717155110256649028,1717155110843921489,1717155111435229286,1717155111941125482,1717155112532429268,1717155113328077026,1717155113904433219,1717155114499110229,1717155114982273645,1717155115609823249,1717155116163627223,1717155116754837894,1717155117291373346,1717155117882903276,1717155118619877682,1717155119157429287,1717155119728037672,1717155120234287411,1717155120764443819,1717155121369687170,1717155121931402462,1717155122541978640,1717155123230410522,1717155123937936354,1717155124549231201,1717155125075503191,1717155125654957547,1717155126209112743,1717155126744369793,1717155127247337019,1717155127826706921,1717155128503523358,1717155129077902351,1717155129657455508,1717155130208268843,1717155130847938035,1717155131472135437,1717155131993181392,1717155132591619426,1717155133318041711,1717155133917896723,1717155134484126245,1717155135012001650,1717155135623295777,1717155136161764066,1717155136711048249,1717155137278700695,1717155137878122768,1717155138605800303],"NumGC":1078723,"NumForcedGC":0,"GCCPUFraction":0.03638902488791019,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437829952,"Frees":437823170},{"Size":16,"Mallocs":6776488074,"Frees":6776121309},{"Size":24,"Mallocs":19163065515,"Frees":19162910672},{"Size":32,"Mallocs":10372071739,"Frees":10371983662},{"Size":48,"Mallocs":1991906302,"Frees":1991856960},{"Size":64,"Mallocs":240742616,"Frees":240709388},{"Size":80,"Mallocs":238986173,"Frees":238970918},{"Size":96,"Mallocs":916562919155,"Frees":916561711405},{"Size":112,"Mallocs":383651508,"Frees":383647537},{"Size":128,"Mallocs":9683436602,"Frees":9683421872},{"Size":144,"Mallocs":190401739,"Frees":190397338},{"Size":160,"Mallocs":71607135,"Frees":71605145},{"Size":176,"Mallocs":5498989,"Frees":5498182},{"Size":192,"Mallocs":32398049,"Frees":32397591},{"Size":208,"Mallocs":68300078,"Frees":68296066},{"Size":224,"Mallocs":3200398,"Frees":3199870},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52414641,"Frees":52387074},{"Size":288,"Mallocs":98659423,"Frees":98636257},{"Size":320,"Mallocs":49989377,"Frees":49985888},{"Size":352,"Mallocs":15148211,"Frees":15147607},{"Size":384,"Mallocs":30460722,"Frees":30460535},{"Size":416,"Mallocs":5930687,"Frees":5928186},{"Size":448,"Mallocs":5902243,"Frees":5897935},{"Size":480,"Mallocs":642394,"Frees":642042},{"Size":512,"Mallocs":39631655,"Frees":39631395},{"Size":576,"Mallocs":21114255,"Frees":21111813},{"Size":640,"Mallocs":1233500,"Frees":1233255},{"Size":704,"Mallocs":18228289,"Frees":18227575},{"Size":768,"Mallocs":6983716,"Frees":6983380},{"Size":896,"Mallocs":7324812,"Frees":7324403},{"Size":1024,"Mallocs":52588067,"Frees":52587838},{"Size":1152,"Mallocs":24989139,"Frees":24989056},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16263609,"Frees":16263580},{"Size":1536,"Mallocs":1801325,"Frees":1801321},{"Size":1792,"Mallocs":12389379,"Frees":12387278},{"Size":2048,"Mallocs":710394,"Frees":710295},{"Size":2304,"Mallocs":22960230,"Frees":22959695},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2038248,"Frees":2038231},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8469646,"Frees":8469623},{"Size":4096,"Mallocs":3539252,"Frees":3539160},{"Size":4864,"Mallocs":8031164,"Frees":8031123},{"Size":5376,"Mallocs":1749695,"Frees":1748151},{"Size":6144,"Mallocs":1312108,"Frees":1312098},{"Size":6528,"Mallocs":6067923,"Frees":6067909},{"Size":6784,"Mallocs":1760710,"Frees":1760634},{"Size":6912,"Mallocs":249988,"Frees":249988},{"Size":8192,"Mallocs":226194,"Frees":226092},{"Size":9472,"Mallocs":1840473,"Frees":1840447},{"Size":9728,"Mallocs":106783,"Frees":106783},{"Size":10240,"Mallocs":1189080,"Frees":1189078},{"Size":10880,"Mallocs":33586,"Frees":33586},{"Size":12288,"Mallocs":273168,"Frees":273164},{"Size":13568,"Mallocs":7828322,"Frees":7828146},{"Size":14336,"Mallocs":2292,"Frees":2268},{"Size":16384,"Mallocs":1390086,"Frees":1390055},{"Size":18432,"Mallocs":1191842,"Frees":1191832}]} 2024-05-31T11:34:14.401Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:34:14.401Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:34:14.401Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:34:14.409Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:34:14.501Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:34:14.507Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:34:14.549Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:34:14.633Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:15.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:15.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:16.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:16.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:16.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:16.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:17.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:17.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:18.308Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:18.534Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:19.338Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:19.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:20.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:20.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:21.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:21.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:22.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:22.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:23.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155262493988216, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:34:23.392Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:23.392Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:23.392Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:34:23.392Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:34:23.392Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=406.781µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:34:23.394Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:23.644Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:24.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:34:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:34:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:34:24.479Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:25.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:25.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:26.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:26.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:26.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:26.724Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:27.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:27.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:28.244Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:28.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:29.324Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:34:29.453Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:34:29.489Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:34:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:34:29.507Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:34:29.507Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:34:29.507Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:34:29.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:29.527Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:34:30.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:30.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:31.134Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155270190340344, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:34:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:31.135Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:31.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:34:31.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:34:31.137Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=568.257µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:34:31.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:31.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:32.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:32.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:33.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:33.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:34.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:34.398Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:34:34.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:34:34.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:34:34.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:35.155Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:35.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:36.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:36.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:36.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:36.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:37.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:37.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:38.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:38.590Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:39.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:39.444Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:40.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:40.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:41.300Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:41.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:42.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:42.600Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:43.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:43.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:44.237Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:34:44.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:34:44.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:34:44.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:34:44.426Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:34:44.501Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:34:44.521Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:34:44.522Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:34:44.522Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:34:44.541Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:34:44.541Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:34:44.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:45.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:45.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:46.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:46.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:46.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:46.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:47.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:47.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:48.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:48.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:49.329Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:49.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:50.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:50.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:51.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:51.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:52.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:52.623Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:53.338Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:53.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155292623549762, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:34:53.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:53.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:34:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:34:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:34:53.391Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=377.392µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:34:53.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:53.745Z ERRO GOXDCR.P2PManager: Request type PeriodicPush to 10.0.9.103:8091 with opaque 903806976 timed out (chan 0xc006baa6b8) 2024-05-31T11:34:54.167Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:54.421Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:34:54.421Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:34:54.422Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:34:54.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:55.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:55.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:56.307Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:56.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:56.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:34:56.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:57.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:57.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:58.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:58.637Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:59.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:34:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:34:59.469Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:34:59.522Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:34:59.523Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:34:59.556Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:35:00.294Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:00.448Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:01.150Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155300295243261, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:35:01.152Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:01.152Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:01.153Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:35:01.166Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:35:01.195Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=623.298µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:35:01.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:01.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:02.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:02.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:03.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:03.649Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:04.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:04.404Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:35:04.404Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:35:04.404Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:35:04.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:05.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:05.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:06.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:06.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:06.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:06.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:07.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:07.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:08.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:08.627Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:10.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:10.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:11.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:11.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:12.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:12.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:13.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:13.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:14.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:14.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:14.398Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:35:14.398Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:35:14.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:35:14.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:35:14.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:35:14.428Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:35:14.500Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:35:14.503Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:35:14.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:35:14.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:35:14.504Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:35:15.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:15.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:16.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:16.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:16.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:16.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:17.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:17.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:18.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:18.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:19.088Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:19.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:20.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:20.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:21.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:21.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:22.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:22.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:23.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:23.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155323360024396, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:35:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:35:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:35:23.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=363.934µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:35:24.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:24.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:24.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:35:24.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:35:24.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:35:25.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:25.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:26.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:26.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:26.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:26.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:27.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:27.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:28.093Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:28.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:29.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:29.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:29.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:35:29.413Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:35:29.428Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:35:29.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:35:29.497Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:35:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:35:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:35:29.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:35:30.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:30.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:31.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:31.134Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155331070254448, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:35:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:35:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:35:31.135Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=515.074µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:35:31.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:32.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:32.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:33.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:33.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:34.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:34.382Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:35:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:35:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:35:35.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:35.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:36.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:36.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:36.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:36.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:37.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:37.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:38.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:38.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:39.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:39.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:40.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:40.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:41.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:41.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:42.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:42.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:43.099Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:43.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:44.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:44.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:35:44.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:35:44.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:35:44.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:35:44.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:35:44.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:35:44.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:35:45.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:45.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:46.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:46.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:46.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:46.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:47.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:47.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:48.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:48.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:49.090Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:49.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:50.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:50.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:51.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:51.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:52.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:52.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:53.088Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:53.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155353369989255, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:35:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:35:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:35:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:35:53.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=402.813µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:35:54.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:54.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:35:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:35:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:35:55.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:55.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:56.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:56.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:56.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:35:56.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:57.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:57.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:58.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:58.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:59.093Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:35:59.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:35:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:35:59.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:35:59.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:35:59.494Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:36:00.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:00.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:01.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155361076326203, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:36:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:36:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:36:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=502.013µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:36:01.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:02.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:02.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:03.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:03.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:04.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:04.384Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:36:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:36:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:36:05.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:05.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:06.101Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:06.330Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:06.331Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:06.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:07.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:07.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:08.135Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:08.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:09.136Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:09.421Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:10.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:10.392Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:11.115Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:12.095Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:12.384Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:13.130Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:13.413Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:14.305Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:14.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:36:14.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:36:14.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:36:14.408Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:36:14.408Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":134930416,"TotalAlloc":91224385274816,"Sys":327718216,"Lookups":0,"Mallocs":971754063523,"Frees":971752711248,"HeapAlloc":134930416,"HeapSys":308510720,"HeapIdle":151183360,"HeapInuse":157327360,"HeapReleased":80863232,"HeapObjects":1352275,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":2524160,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193682,"GCSys":7331136,"OtherSys":1609542,"NextGC":215111984,"LastGC":1717155374242976746,"PauseTotalNs":3785418875642,"PauseNs":[174050,309926,122696,134517,142781,137319,143201,127204,123502,132145,139141,142673,136605,133231,140319,127201,148634,131710,136645,119612,132778,137647,132918,137226,147437,138316,145460,137079,126649,124036,140308,125523,129367,136049,126230,126929,169583,126005,135866,115571,123998,132426,142791,111629,135256,298873,139939,134122,161320,138781,178243,133293,147428,126533,302586,138410,131283,138279,136158,140351,123910,132366,138754,208006,154596,131917,134882,134711,132563,127768,138176,126728,117331,123058,161531,3645861,114079,141143,124529,133155,783780,163426,139446,119220,138522,122341,118722,141427,137582,126068,139042,137557,167121,116488,144526,134119,147003,144095,124323,132648,185700,163715,154311,141362,126703,125966,142713,155013,125821,144957,132245,134593,123190,129289,134715,126701,132357,131666,139624,117670,196880,124029,139099,137846,139762,127710,142737,146904,131000,135395,120714,126447,143190,135013,141605,125953,113462,132319,122783,139790,302100,139454,131774,138059,126366,131063,139875,121581,125155,121316,143503,124418,139794,301828,135539,134060,1519000,133330,108560,144686,141357,132080,132782,130674,150327,119708,97304,122722,129846,135350,136544,159721,138323,118703,113362,148969,147264,128915,132314,119401,142648,136527,138451,144833,139523,153792,151892,142498,137577,146878,150213,328591,133723,115606,143195,139269,136148,142024,152711,138057,139290,125368,150580,138691,142586,301890,141455,132701,211175,318371,277706,140666,129622,136045,139928,134050,115022,141485,133796,3524225,113527,293646,152487,207387,132751,130219,138035,203043,121552,1023697,132068,142263,124483,135855,112059,124767,90033,8133440,134216,134379,133108,136095,136088,137799,125990,138352,144676,8508490,4042476,10829391,119386,132523,138545,131531,158693,131362],"PauseEnd":[1717155373304602989,1717155373537119446,1717155373717407375,1717155373884795326,1717155374242976746,1717155338578828593,1717155338708371921,1717155338830566123,1717155338952395881,1717155339090626397,1717155339218435288,1717155339356365744,1717155339496368477,1717155339630235397,1717155339759883227,1717155339889766248,1717155340013555907,1717155340133050069,1717155340263716361,1717155340397155382,1717155340527330503,1717155340651941090,1717155340776143703,1717155340906266541,1717155341039180057,1717155341160473599,1717155341288513358,1717155341411102456,1717155341536483232,1717155341659010089,1717155341778028798,1717155341900730197,1717155342028339997,1717155342151904226,1717155342283088262,1717155342408998968,1717155342538945717,1717155342667841297,1717155342796836428,1717155342927755940,1717155343070152892,1717155343211730503,1717155343347217224,1717155343473230551,1717155343602546629,1717155343732473217,1717155343863622051,1717155343992967194,1717155344122051398,1717155344249514406,1717155344382673926,1717155344514141255,1717155344646900422,1717155344781267331,1717155344909738910,1717155345041566934,1717155345168194291,1717155345299051899,1717155345418744028,1717155345543110608,1717155345672900856,1717155345802700783,1717155345934368909,1717155346083272817,1717155346217054947,1717155346342366945,1717155346461117460,1717155346587635192,1717155346711992276,1717155346830849604,1717155346954235175,1717155347095756222,1717155347230082599,1717155347373044395,1717155347502972966,1717155347634452327,1717155347757784732,1717155347879064616,1717155348007553333,1717155348128107180,1717155348258446711,1717155348384862607,1717155348501705913,1717155348624135447,1717155348747679241,1717155348862656493,1717155348975360221,1717155349107018556,1717155349241078424,1717155349379299344,1717155349512309301,1717155349631874823,1717155349753920392,1717155349883356789,1717155350012502412,1717155350129531219,1717155350253066314,1717155350391001991,1717155350522253658,1717155350648996904,1717155350779348467,1717155350900373280,1717155351024753448,1717155351149501834,1717155351274403016,1717155351393113293,1717155351523993696,1717155351650925383,1717155351773505613,1717155351895674041,1717155352010356908,1717155352131079232,1717155352254409385,1717155352377292631,1717155352502146180,1717155352626815433,1717155352760111614,1717155352892092863,1717155353059101144,1717155353199356980,1717155353334987637,1717155353469255204,1717155353604005710,1717155353740443939,1717155353870795290,1717155353991041537,1717155354118561236,1717155354251389237,1717155354385696275,1717155354517143747,1717155354647454318,1717155354770813253,1717155354895094183,1717155355025457163,1717155355147880433,1717155355273485023,1717155355400999191,1717155355530539947,1717155355659261533,1717155355788695185,1717155355921450309,1717155356065713119,1717155356200166164,1717155356330568809,1717155356451465997,1717155356578039075,1717155356707792027,1717155356834538987,1717155356956727396,1717155357094226603,1717155357222036792,1717155357359781064,1717155357492741051,1717155357625118695,1717155357755375173,1717155357883048383,1717155358014926343,1717155358145194411,1717155358287414208,1717155358417530249,1717155358548849713,1717155358679070916,1717155358808531272,1717155358936636436,1717155359092187627,1717155359231304397,1717155359377757066,1717155359519392464,1717155359649914580,1717155359773088940,1717155359899654286,1717155360033065739,1717155360167796962,1717155360300658309,1717155360429998968,1717155360563780470,1717155360694960982,1717155360815398173,1717155360941114136,1717155361089092717,1717155361226532443,1717155361373428375,1717155361515884060,1717155361640753228,1717155361765184366,1717155361898283183,1717155362032928739,1717155362161751341,1717155362290321959,1717155362419041505,1717155362551590123,1717155362681958947,1717155362813277879,1717155362942613825,1717155363106745436,1717155363245278551,1717155363381961782,1717155363515950428,1717155363654979871,1717155363781046992,1717155363908513620,1717155364045159680,1717155364193453740,1717155364336267454,1717155364496380102,1717155364633891934,1717155364761530336,1717155364904486103,1717155365048689536,1717155365189501906,1717155365330288374,1717155365481315629,1717155365629482651,1717155365772592803,1717155365914876424,1717155366056017761,1717155366192829834,1717155366328221558,1717155366470444372,1717155366614334743,1717155366750284836,1717155366888949253,1717155367016699907,1717155367171202491,1717155367312552421,1717155367471803066,1717155367617325727,1717155367760056025,1717155367897021307,1717155368046354252,1717155368205792376,1717155368383040206,1717155368541160305,1717155368676727009,1717155368817892888,1717155368952735446,1717155369163751107,1717155369394001666,1717155369606394339,1717155369799211338,1717155369988136381,1717155370219684580,1717155370457169111,1717155370645264840,1717155370839021623,1717155371022289522,1717155371242359751,1717155371442912611,1717155371646020896,1717155371846205352,1717155372037344292,1717155372237661661,1717155372465313274,1717155372659692941,1717155372850627808,1717155373045882009],"NumGC":1079301,"NumForcedGC":0,"GCCPUFraction":0.036397856540531744,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":437939799,"Frees":437933091},{"Size":16,"Mallocs":6778289736,"Frees":6777927431},{"Size":24,"Mallocs":19168210889,"Frees":19168053219},{"Size":32,"Mallocs":10374832901,"Frees":10374741905},{"Size":48,"Mallocs":1992457467,"Frees":1992411166},{"Size":64,"Mallocs":240786421,"Frees":240755247},{"Size":80,"Mallocs":239027171,"Frees":239011928},{"Size":96,"Mallocs":917093625930,"Frees":917093087254},{"Size":112,"Mallocs":383742695,"Frees":383738729},{"Size":128,"Mallocs":9686060935,"Frees":9686042524},{"Size":144,"Mallocs":190445365,"Frees":190440973},{"Size":160,"Mallocs":71617602,"Frees":71615613},{"Size":176,"Mallocs":5499580,"Frees":5498780},{"Size":192,"Mallocs":32407172,"Frees":32406711},{"Size":208,"Mallocs":68316412,"Frees":68312416},{"Size":224,"Mallocs":3200721,"Frees":3200195},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52423891,"Frees":52396328},{"Size":288,"Mallocs":98665601,"Frees":98646066},{"Size":320,"Mallocs":49995948,"Frees":49992468},{"Size":352,"Mallocs":15151231,"Frees":15150668},{"Size":384,"Mallocs":30469571,"Frees":30469381},{"Size":416,"Mallocs":5932055,"Frees":5929564},{"Size":448,"Mallocs":5902269,"Frees":5897962},{"Size":480,"Mallocs":642394,"Frees":642042},{"Size":512,"Mallocs":39642708,"Frees":39642469},{"Size":576,"Mallocs":21115609,"Frees":21113176},{"Size":640,"Mallocs":1233756,"Frees":1233511},{"Size":704,"Mallocs":18232711,"Frees":18231997},{"Size":768,"Mallocs":6985852,"Frees":6985513},{"Size":896,"Mallocs":7326874,"Frees":7326465},{"Size":1024,"Mallocs":52601697,"Frees":52601486},{"Size":1152,"Mallocs":24991735,"Frees":24991660},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16268013,"Frees":16267984},{"Size":1536,"Mallocs":1801819,"Frees":1801815},{"Size":1792,"Mallocs":12393119,"Frees":12391017},{"Size":2048,"Mallocs":710575,"Frees":710477},{"Size":2304,"Mallocs":22962256,"Frees":22961731},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2038641,"Frees":2038624},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8472515,"Frees":8472489},{"Size":4096,"Mallocs":3540018,"Frees":3539931},{"Size":4864,"Mallocs":8031913,"Frees":8031876},{"Size":5376,"Mallocs":1749763,"Frees":1748219},{"Size":6144,"Mallocs":1312132,"Frees":1312122},{"Size":6528,"Mallocs":6070055,"Frees":6070038},{"Size":6784,"Mallocs":1761447,"Frees":1761371},{"Size":6912,"Mallocs":250048,"Frees":250048},{"Size":8192,"Mallocs":226243,"Frees":226141},{"Size":9472,"Mallocs":1841098,"Frees":1841075},{"Size":9728,"Mallocs":106831,"Frees":106831},{"Size":10240,"Mallocs":1189320,"Frees":1189318},{"Size":10880,"Mallocs":33597,"Frees":33597},{"Size":12288,"Mallocs":273237,"Frees":273233},{"Size":13568,"Mallocs":7831191,"Frees":7831031},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1390479,"Frees":1390448},{"Size":18432,"Mallocs":1192283,"Frees":1192277}]} 2024-05-31T11:36:14.435Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:36:14.489Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:36:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:36:14.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:15.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:15.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:16.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:16.400Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:16.400Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:16.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:17.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:17.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:18.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:18.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:19.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:19.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:20.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:20.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:21.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:21.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:22.156Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:22.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:23.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155382585752569, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:36:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:23.418Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:36:23.418Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:36:23.419Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=374.862µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:36:23.584Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:24.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:24.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:36:24.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:36:24.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:36:24.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:25.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:25.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:26.160Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:26.404Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:26.404Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:26.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:27.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:27.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:28.298Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:28.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:29.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:29.407Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:36:29.468Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:36:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:36:29.512Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:36:29.538Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:30.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:30.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:31.142Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155390165424168, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:36:31.143Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:31.144Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:31.144Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:36:31.145Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:36:31.146Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=618.514µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:36:31.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:31.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:32.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:32.443Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:33.326Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:33.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:34.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:34.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:36:34.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:36:34.404Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:36:34.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:35.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:35.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:36.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:36.391Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:36.392Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:36.627Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:37.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:37.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:38.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:38.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:39.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:39.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:40.160Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:40.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:41.233Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:41.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:42.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:42.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:43.283Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:43.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:44.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:44.404Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:36:44.443Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:36:44.443Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:36:44.443Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:36:44.497Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:36:44.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:36:44.550Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:36:44.621Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:45.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:45.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:46.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:46.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:46.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:46.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:47.334Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:47.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:48.410Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:48.608Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:49.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:49.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:50.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:50.439Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:51.339Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:51.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:52.298Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:52.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:53.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:53.410Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155412526254093, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:36:53.411Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:53.411Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:36:53.411Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:36:53.411Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:36:53.411Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=384.802µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:36:53.604Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:54.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:54.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:36:54.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:36:54.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:36:54.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:55.306Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:55.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:56.162Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:56.331Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:56.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:36:56.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:57.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:57.534Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:58.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:58.603Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:36:59.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:36:59.401Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:36:59.435Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:36:59.482Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:36:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:36:59.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:00.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:00.511Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155420206201659, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:37:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:37:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:37:01.135Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=601.382µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:37:01.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:01.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:02.210Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:02.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:03.341Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:03.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:04.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:04.414Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:37:04.414Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:37:04.414Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:37:04.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:05.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:05.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:06.193Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:06.387Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:06.387Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:06.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:07.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:07.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:08.333Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:08.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:09.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:09.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:10.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:10.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:11.237Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:11.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:12.285Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:12.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:13.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:13.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:14.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:14.416Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:37:14.417Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:37:14.417Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:37:14.417Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:37:14.504Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:37:14.522Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:37:14.549Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:37:14.549Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:37:14.549Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:37:14.557Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:37:14.574Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:37:14.629Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:15.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:15.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:16.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:16.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:16.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:16.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:17.332Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:17.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:18.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:18.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:19.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:19.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:20.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:20.459Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:21.348Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:21.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:22.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:22.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:23.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155442540770982, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:37:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:37:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:37:23.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=377.712µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:37:23.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:24.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:37:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:37:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:37:24.511Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:25.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:25.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:26.170Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:26.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:26.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:26.675Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:27.305Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:27.479Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:28.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:28.656Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:29.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:29.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:37:29.437Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:37:29.497Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:37:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:37:29.532Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:37:29.532Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:37:29.532Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:37:29.534Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:37:29.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:30.326Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:30.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:31.145Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155450326702098, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:37:31.145Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:31.145Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:31.145Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:37:31.146Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:37:31.146Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=520.041µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:37:31.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:31.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:32.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:32.501Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:33.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:33.655Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:34.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:34.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:37:34.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:37:34.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:37:34.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:35.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:35.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:36.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:36.355Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:36.355Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:36.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:37.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:37.464Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:38.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:38.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:39.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:39.656Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:40.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:40.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:41.336Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:41.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:42.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:42.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:43.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:43.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:44.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:44.414Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:37:44.415Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:37:44.416Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:37:44.416Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:37:44.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:37:44.509Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:37:44.577Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:37:44.657Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:45.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:45.459Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:46.256Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:46.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:46.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:46.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:47.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:47.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:48.294Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:48.618Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:49.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:49.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:50.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:50.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:51.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:51.658Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:52.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:52.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:53.308Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:53.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155472469405031, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:37:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:37:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:37:53.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:37:53.388Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=414.447µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:37:53.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:54.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:37:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:37:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:37:54.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:55.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:55.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:56.331Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:56.403Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:56.404Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:37:56.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:57.156Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:57.553Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:58.326Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:58.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:59.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:37:59.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:37:59.426Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:37:59.471Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:37:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:37:59.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:37:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:37:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:37:59.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:37:59.518Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:38:00.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:00.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155480166582698, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:38:01.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:01.135Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:01.135Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:38:01.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:38:01.137Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=642.382µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:38:01.225Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:01.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:02.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:02.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:03.273Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:03.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:04.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:38:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:38:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:38:04.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:05.317Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:05.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:06.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:06.381Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:06.381Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:06.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:07.147Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:07.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:08.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:08.652Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:09.305Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:09.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:10.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:10.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:11.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:11.621Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:12.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:12.441Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:13.408Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:13.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:14.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:14.422Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":193880640,"TotalAlloc":91245316823640,"Sys":327718216,"Lookups":0,"Mallocs":971978456383,"Frees":971976498525,"HeapAlloc":193880640,"HeapSys":308379648,"HeapIdle":92594176,"HeapInuse":215785472,"HeapReleased":68100096,"HeapObjects":1957858,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3665280,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193682,"GCSys":7462208,"OtherSys":1609542,"NextGC":203627440,"LastGC":1717155493913534034,"PauseTotalNs":3787797738191,"PauseNs":[174050,309926,122696,134517,142781,22265452,18387877,13146083,15052499,5107001,122095,19966503,16293658,231966,3692126,18486387,4309075,16247385,11450379,19000826,11562017,19413647,3996248,142410,10219470,15046864,9254500,14590689,12189578,19029029,311953,21476868,8169246,11738127,13616008,29819043,2533608,11721127,147913,21916736,7799335,136533,11508960,7911130,4099875,12275385,6486607,22734531,7853206,19064982,126147,16125732,16131484,24812795,19764558,10870733,27539863,128532,8921202,9820379,12463603,10947117,12119328,133928,8024758,25161875,24915231,148390,16651349,21478505,6196614,2687884,8117324,4895873,25772047,14859981,8120144,19914910,7879115,4211483,16165692,695996,10888206,6959215,13750688,20066016,17319370,16336673,27945180,15163020,107643,10416583,118088,29100471,230920,14249479,6562476,15916336,13224164,10389965,22210197,10801043,290018,7272776,7670918,16534105,13191166,14357577,8099142,18547778,11600921,28474932,3797611,21877263,136317,17846350,23404870,4841145,24482239,5316206,22148936,145255,8080308,24518112,11415856,3987904,12152333,16202945,12175948,21579983,147625,12159855,12601022,20979820,22114065,12136430,15263018,13529242,21651486,4725190,96985,5242522,8020466,11338520,15551295,5310217,138492,100396,15597660,18536604,25783803,12221802,16946626,9110944,12209576,11031296,332907,91399,141636,9481343,2424448,13749583,13996545,12121383,8021313,144852,9843315,8163562,12705695,14540169,8939473,17461177,186199,4139997,9555181,10650420,3914794,1729113,23477584,12217958,20152851,8486596,1234802,1385954,15234325,25633793,155624,14267723,20172975,16212326,10949549,8934768,102887,7802481,20983732,9453912,141558,16178978,89013,321188,15718516,13130140,12408317,12241152,22212923,156976,131463,22724643,26319294,9456999,7982014,2025525,1438139,6214254,139928,134050,115022,141485,133796,3524225,113527,293646,152487,207387,132751,130219,138035,203043,121552,1023697,132068,142263,124483,135855,112059,124767,90033,8133440,134216,134379,133108,136095,136088,137799,125990,138352,144676,8508490,4042476,10829391,119386,132523,138545,131531,158693,131362],"PauseEnd":[1717155373304602989,1717155373537119446,1717155373717407375,1717155373884795326,1717155374242976746,1717155374835070314,1717155375379917047,1717155375909037485,1717155376397681389,1717155376891560125,1717155377453792718,1717155377992121849,1717155378747904054,1717155379314147283,1717155379852762207,1717155380370927328,1717155380872767250,1717155381427572120,1717155381923287404,1717155382477536280,1717155383017069310,1717155383764389349,1717155384315422788,1717155384857246191,1717155385379163787,1717155385873424644,1717155386476405816,1717155386993146088,1717155387590926712,1717155388189340972,1717155388832435980,1717155389406525183,1717155389961258312,1717155390515081425,1717155391017301030,1717155391640753065,1717155392146837995,1717155392693753448,1717155393208742993,1717155393856247117,1717155394418558099,1717155394913066408,1717155395411421594,1717155395952448057,1717155396537800780,1717155397040311188,1717155397654734645,1717155398185371374,1717155398848845109,1717155399428145574,1717155399923522571,1717155400493690891,1717155401047950243,1717155401674872566,1717155402168632562,1717155402726986621,1717155403427173841,1717155403940021351,1717155404587938839,1717155405130222506,1717155405668271980,1717155406183520274,1717155406738205838,1717155407226826010,1717155407787681669,1717155408407337645,1717155409035263396,1717155409632311617,1717155410160664860,1717155410707100368,1717155411254954379,1717155411773237446,1717155412331330536,1717155412884471226,1717155413574882906,1717155414154866706,1717155414732438723,1717155415266688189,1717155415768511469,1717155416328086387,1717155416850096283,1717155417396441732,1717155417947112743,1717155418626952701,1717155419200638506,1717155419811437221,1717155420337679536,1717155420862483257,1717155421459840626,1717155421949455549,1717155422566891755,1717155423188191961,1717155423860725706,1717155424414052638,1717155424913385659,1717155425448240754,1717155425938505734,1717155426509396189,1717155426984045198,1717155427559842792,1717155428138082230,1717155428791601111,1717155429327221686,1717155429847452382,1717155430490722652,1717155431061189134,1717155431679648497,1717155432202112304,1717155432741335757,1717155433364503987,1717155433942398276,1717155434520771015,1717155435036452847,1717155435647757515,1717155436146696193,1717155436728575706,1717155437295103508,1717155437854774388,1717155438482903440,1717155439082275269,1717155439656027805,1717155440146153447,1717155440730571374,1717155441279915719,1717155441814711577,1717155442394058481,1717155442956808180,1717155443712503704,1717155444262655102,1717155444865064011,1717155445460289611,1717155445993145341,1717155446657835301,1717155447312119792,1717155447895533961,1717155448579563355,1717155449090327387,1717155449669997923,1717155450208499195,1717155450727002977,1717155451285499153,1717155451798919121,1717155452374251603,1717155452908029145,1717155453652036226,1717155454170961694,1717155454722058721,1717155455246282068,1717155455823204696,1717155456381715108,1717155457046089349,1717155457762984309,1717155458484666256,1717155459021912189,1717155459653222217,1717155460175158816,1717155460735678594,1717155461310050280,1717155461801874401,1717155462315218075,1717155462784324707,1717155463466108851,1717155464036852747,1717155464656080187,1717155465183233344,1717155465767045231,1717155466304314221,1717155466837379752,1717155467414938003,1717155467949983606,1717155468672567021,1717155469204136013,1717155469739207744,1717155470313613848,1717155470948584714,1717155471639257293,1717155472194189536,1717155472734167567,1717155473436982616,1717155474048986291,1717155474676825351,1717155475208264414,1717155475780825108,1717155476403155921,1717155476928371169,1717155477523934793,1717155478052549786,1717155478738877027,1717155479305023226,1717155479867357817,1717155480422146644,1717155480942823733,1717155481652998317,1717155482201513978,1717155482766761638,1717155483490369610,1717155484120016173,1717155484706322943,1717155485240950991,1717155485784191324,1717155486374760961,1717155486901649673,1717155487429241549,1717155487999421050,1717155488719800590,1717155489299061048,1717155489834047973,1717155490408767796,1717155490918623081,1717155491573329406,1717155492103131764,1717155492660051849,1717155493240081995,1717155493913534034,1717155365914876424,1717155366056017761,1717155366192829834,1717155366328221558,1717155366470444372,1717155366614334743,1717155366750284836,1717155366888949253,1717155367016699907,1717155367171202491,1717155367312552421,1717155367471803066,1717155367617325727,1717155367760056025,1717155367897021307,1717155368046354252,1717155368205792376,1717155368383040206,1717155368541160305,1717155368676727009,1717155368817892888,1717155368952735446,1717155369163751107,1717155369394001666,1717155369606394339,1717155369799211338,1717155369988136381,1717155370219684580,1717155370457169111,1717155370645264840,1717155370839021623,1717155371022289522,1717155371242359751,1717155371442912611,1717155371646020896,1717155371846205352,1717155372037344292,1717155372237661661,1717155372465313274,1717155372659692941,1717155372850627808,1717155373045882009],"NumGC":1079510,"NumForcedGC":0,"GCCPUFraction":0.03640959535179988,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438057946,"Frees":438051191},{"Size":16,"Mallocs":6780275271,"Frees":6779909630},{"Size":24,"Mallocs":19173332655,"Frees":19173178354},{"Size":32,"Mallocs":10377623335,"Frees":10377535775},{"Size":48,"Mallocs":1993052228,"Frees":1993003961},{"Size":64,"Mallocs":240831505,"Frees":240799296},{"Size":80,"Mallocs":239073353,"Frees":239058108},{"Size":96,"Mallocs":917302880133,"Frees":917301734523},{"Size":112,"Mallocs":383833886,"Frees":383829916},{"Size":128,"Mallocs":9688677324,"Frees":9688662611},{"Size":144,"Mallocs":190500248,"Frees":190495850},{"Size":160,"Mallocs":71628067,"Frees":71626078},{"Size":176,"Mallocs":5500144,"Frees":5499338},{"Size":192,"Mallocs":32415989,"Frees":32415531},{"Size":208,"Mallocs":68332255,"Frees":68328244},{"Size":224,"Mallocs":3201046,"Frees":3200518},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52433136,"Frees":52405566},{"Size":288,"Mallocs":98692227,"Frees":98670086},{"Size":320,"Mallocs":50002527,"Frees":49999042},{"Size":352,"Mallocs":15153855,"Frees":15153262},{"Size":384,"Mallocs":30478111,"Frees":30477923},{"Size":416,"Mallocs":5933436,"Frees":5930939},{"Size":448,"Mallocs":5902295,"Frees":5897987},{"Size":480,"Mallocs":642394,"Frees":642042},{"Size":512,"Mallocs":39653469,"Frees":39653208},{"Size":576,"Mallocs":21122077,"Frees":21119635},{"Size":640,"Mallocs":1234015,"Frees":1233770},{"Size":704,"Mallocs":18237132,"Frees":18236418},{"Size":768,"Mallocs":6987690,"Frees":6987354},{"Size":896,"Mallocs":7328860,"Frees":7328451},{"Size":1024,"Mallocs":52615350,"Frees":52615120},{"Size":1152,"Mallocs":24999441,"Frees":24999357},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16272418,"Frees":16272389},{"Size":1536,"Mallocs":1802318,"Frees":1802314},{"Size":1792,"Mallocs":12396485,"Frees":12394385},{"Size":2048,"Mallocs":710740,"Frees":710641},{"Size":2304,"Mallocs":22969382,"Frees":22968848},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2039036,"Frees":2039019},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8475003,"Frees":8474980},{"Size":4096,"Mallocs":3540783,"Frees":3540694},{"Size":4864,"Mallocs":8033668,"Frees":8033626},{"Size":5376,"Mallocs":1749831,"Frees":1748287},{"Size":6144,"Mallocs":1312157,"Frees":1312147},{"Size":6528,"Mallocs":6071890,"Frees":6071876},{"Size":6784,"Mallocs":1762100,"Frees":1762024},{"Size":6912,"Mallocs":250108,"Frees":250108},{"Size":8192,"Mallocs":226291,"Frees":226189},{"Size":9472,"Mallocs":1841707,"Frees":1841683},{"Size":9728,"Mallocs":106877,"Frees":106877},{"Size":10240,"Mallocs":1189560,"Frees":1189558},{"Size":10880,"Mallocs":33609,"Frees":33609},{"Size":12288,"Mallocs":273305,"Frees":273301},{"Size":13568,"Mallocs":7833679,"Frees":7833507},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1390871,"Frees":1390840},{"Size":18432,"Mallocs":1192716,"Frees":1192708}]} 2024-05-31T11:38:14.422Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:38:14.423Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:38:14.423Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:38:14.425Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:38:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:38:14.508Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:38:14.509Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:38:14.509Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:38:14.509Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:38:14.547Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:38:14.576Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:38:14.639Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:15.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:15.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:16.335Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:16.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:16.343Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:16.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:17.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:17.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:18.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:18.613Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:19.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:19.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:20.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:20.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:21.310Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:21.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:22.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:22.511Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:23.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:23.389Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155502511796517, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:38:23.390Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:23.390Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:23.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:38:23.392Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:38:23.405Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=12.730272ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:38:23.658Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:24.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:38:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:38:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:38:24.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:25.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:25.522Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:26.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:26.356Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:26.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:26.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:27.171Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:27.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:28.229Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:28.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:29.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:29.398Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:38:29.448Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:38:29.488Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:38:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:38:29.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:30.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:30.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:31.135Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155510164272618, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:38:31.135Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:31.135Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:31.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:38:31.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:38:31.136Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=502.595µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:38:31.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:31.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:32.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:32.556Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:33.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:33.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:34.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:38:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:38:34.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:38:34.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:35.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:35.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:36.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:36.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:36.372Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:36.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:37.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:37.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:38.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:38.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:39.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:39.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:40.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:40.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:41.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:41.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:42.254Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:42.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:43.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:43.611Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:44.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:44.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:38:44.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:38:44.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:38:44.402Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:38:44.469Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:38:44.523Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:38:44.551Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:38:44.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:45.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:45.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:46.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:46.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:46.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:46.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:47.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:47.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:48.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:48.601Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:49.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:49.449Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:50.306Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:50.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:51.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:51.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:52.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:52.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:53.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:53.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155532548766092, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:38:53.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:53.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:38:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:38:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:38:53.392Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=379.291µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:38:53.637Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:54.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:38:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:38:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:38:54.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:55.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:55.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:56.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:56.381Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:56.382Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:38:56.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:57.253Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:57.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:58.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:58.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:38:59.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:38:59.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:38:59.474Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:38:59.515Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:38:59.550Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:38:59.638Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:00.224Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:00.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:01.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155540224607798, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:39:01.187Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:01.188Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:01.189Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:39:01.189Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:39:01.190Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=643.187µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:39:01.430Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:01.640Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:02.334Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:02.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:03.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:03.672Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:04.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:04.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:39:04.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:39:04.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:39:04.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:05.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:05.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:06.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:06.357Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:06.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:06.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:07.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:07.663Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:08.403Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:08.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:09.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:09.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:10.305Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:10.457Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:11.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:11.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:12.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:12.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:13.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:13.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:14.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:14.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:39:14.409Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:39:14.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:39:14.410Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:39:14.442Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:39:14.486Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:39:14.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:14.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:39:14.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:39:14.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:39:14.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:39:14.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:39:15.333Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:15.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:16.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:16.396Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:16.402Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:16.615Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:17.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:17.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:18.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:18.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:19.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:19.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:20.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:20.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:21.343Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:21.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:22.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:22.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:23.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:23.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155562555240480, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:39:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:23.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:39:23.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:39:23.388Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=372.978µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:39:23.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:24.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:24.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:39:24.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:39:24.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:39:24.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:25.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:25.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:26.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:26.362Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:26.364Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:26.582Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:27.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:27.588Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:28.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:28.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:29.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:39:29.464Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:39:29.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:39:29.499Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:39:29.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:30.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:30.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:31.153Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155570174044253, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:39:31.153Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:31.153Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:31.153Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:39:31.153Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:39:31.153Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=489.841µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:39:31.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:31.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:32.326Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:32.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:33.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:33.682Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:34.267Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:34.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:39:34.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:39:34.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:39:34.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:35.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:35.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:36.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:36.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:36.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:37.405Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:37.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:38.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:38.807Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:39.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:39.553Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:40.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:40.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:41.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:41.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:42.210Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:42.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:43.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:43.669Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:44.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:44.398Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:39:44.399Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:39:44.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:39:44.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:39:44.454Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:39:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:39:44.526Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:39:44.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:45.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:45.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:46.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:46.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:46.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:46.432Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:47.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:47.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:48.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:48.631Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:49.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:49.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:50.333Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:50.501Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:51.317Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:51.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:52.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:52.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:53.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:53.418Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155592563829636, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:39:53.418Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:53.418Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:39:53.418Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:39:53.418Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:39:53.419Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=422.484µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:39:53.699Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:54.253Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:54.405Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:39:54.405Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:39:54.405Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:39:54.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:55.157Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:55.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:56.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:56.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:56.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:39:56.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:57.313Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:57.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:58.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:58.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:59.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:39:59.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:39:59.449Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:39:59.451Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:39:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:39:59.497Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:39:59.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:39:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:39:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:39:59.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:40:00.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:00.503Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155600358009633, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:40:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:40:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:40:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=492.116µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:40:01.196Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:01.582Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:02.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:02.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:03.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:03.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:04.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:04.424Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:40:04.424Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:40:04.424Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:40:04.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:05.225Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:05.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:06.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:06.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:06.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:06.438Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:07.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:07.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:08.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:08.624Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:10.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:10.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:11.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:11.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:12.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:12.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:13.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:13.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:14.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:14.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:14.396Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":113808384,"TotalAlloc":91268654241864,"Sys":327718216,"Lookups":0,"Mallocs":972227849335,"Frees":972226727826,"HeapAlloc":113808384,"HeapSys":308510720,"HeapIdle":172228608,"HeapInuse":136282112,"HeapReleased":85426176,"HeapObjects":1121509,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":2146400,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193930,"GCSys":7331136,"OtherSys":1609294,"NextGC":211779728,"LastGC":1717155614383964278,"PauseTotalNs":3790161069481,"PauseNs":[12171550,13296494,15234274,10048182,98952,8937100,25777925,16453723,7651451,15894336,15686508,125940,12209318,8052861,17974832,16714613,13131613,13332460,24569509,13980995,15558831,139234,131966,14856909,11912419,4097250,6411096,5228240,18684525,140584,15264472,18988011,29013388,1195261,8498936,4277745,16134816,88012,134887,12528453,39558194,31856883,16544989,21699071,19848535,12108909,7235821,4121021,3705048,153134,15887327,27450213,20194312,10120143,4083665,21599543,211539,15996489,10320224,137381,7152027,12027000,154717,140685,8521534,16132887,21728883,19870208,16157326,15706435,15810984,22559998,19722558,9734572,20151333,7672805,8248009,16179728,27889325,131202,11892393,8119588,15695752,12029182,4645331,10986480,11600313,18788490,3812028,14959838,17547570,6825207,4070729,11765084,9472428,7875664,17875810,4080186,12772359,9822463,145550,30501797,159174,1377536,33659376,17728756,328129,13937516,12117611,22685589,11514681,16197597,16151653,3623908,7639914,24524496,1008533,312970,30679258,19315000,16061177,15939631,9717594,2081348,10420605,150002,8885576,18918612,152961,12159203,12034692,23191773,15973468,172696,9836966,19674527,185072,10240407,8312577,13818536,15570030,21194832,23569638,7947772,17505844,3907660,27620890,22489514,20070768,24638588,13354870,104779,4201351,130286,1528753,127659,127345,109694,123380,128390,115949,125943,141198,130178,132750,122474,136750,148372,137075,157346,133052,139372,130135,294961,130689,123698,296728,116637,290630,128953,126492,133038,130208,135759,336463,138693,128545,132352,139092,132001,125907,155398,118336,7802481,20983732,9453912,141558,16178978,89013,321188,15718516,13130140,12408317,12241152,22212923,156976,131463,22724643,26319294,9456999,7982014,2025525,1438139,6214254,8094012,12067591,24739737,17048820,4113484,13651440,3024089,4306302,3692963,11306469,16277387,19582995,7780619,14212882,30911517,11928503,12439193,11895969,12118211,13278461,12006618,22605329,4102075,3938017,12135952,7153565,115328,16102709,21240587,293262,3658843,148661,22684433,149932,11907292,24720982,10388081,15014042,8132388,16394396,137757,8987490],"PauseEnd":[1717155518185417219,1717155518861727016,1717155519440808426,1717155519947695183,1717155520583275213,1717155521070015836,1717155521672294963,1717155522218980855,1717155522750918068,1717155523405767285,1717155524000892344,1717155524604190583,1717155525086562846,1717155525668831710,1717155526198364843,1717155526740938552,1717155527367343076,1717155527934541857,1717155528678393328,1717155529188999930,1717155529744161824,1717155530269155474,1717155530789399640,1717155531315001075,1717155531906804585,1717155532473976387,1717155532989383969,1717155533736589715,1717155534319303497,1717155534829312801,1717155535412461930,1717155535942916689,1717155536588652957,1717155537104237604,1717155537707111457,1717155538319557978,1717155538953652531,1717155539515879192,1717155540026741732,1717155540616231450,1717155541182963704,1717155542067631886,1717155542905719830,1717155543620680992,1717155544279604825,1717155544849095009,1717155545399635873,1717155545906261616,1717155546481830305,1717155546960305882,1717155547661040230,1717155548374019116,1717155549047082738,1717155549645025567,1717155550196978443,1717155550777857616,1717155551332191546,1717155551903818637,1717155552424753826,1717155552942924919,1717155553668678920,1717155554188498020,1717155554769818557,1717155555314830072,1717155555836219008,1717155556472745176,1717155557013668720,1717155557683117871,1717155558299623918,1717155558944653024,1717155559549707003,1717155560074207922,1717155560699346857,1717155561204039318,1717155561813142573,1717155562405980860,1717155563079061365,1717155563812341896,1717155564389250834,1717155564946955997,1717155565583207015,1717155566180117995,1717155566905266771,1717155567522293001,1717155568107818909,1717155568817901632,1717155569389577093,1717155569934577849,1717155570547807349,1717155571068979121,1717155571694132197,1717155572307311563,1717155572916384650,1717155573681749666,1717155574311371593,1717155574871961248,1717155575425009193,1717155575981930325,1717155576683157872,1717155577404884741,1717155577995118606,1717155578827380246,1717155579398172372,1717155579925483975,1717155580554154539,1717155581119257892,1717155581768402166,1717155582325276361,1717155582906311600,1717155583627638819,1717155584344654076,1717155584913661342,1717155585546585909,1717155586104069135,1717155586640925448,1717155587176968578,1717155587774942445,1717155588462137916,1717155589156766783,1717155589698386104,1717155590196070895,1717155590757969986,1717155591300952470,1717155591874227194,1717155592434052285,1717155592965207620,1717155593683523147,1717155594357288497,1717155594883067942,1717155595501029374,1717155596046922163,1717155596673829896,1717155597208487486,1717155597818894514,1717155598430897995,1717155599125176556,1717155599711042186,1717155600269124855,1717155600854907772,1717155601476471931,1717155602070268920,1717155602671969065,1717155603213056789,1717155603898904913,1717155604519858518,1717155605082906462,1717155605711180377,1717155606168194970,1717155606745279649,1717155607370941078,1717155607947957297,1717155608657583307,1717155609065227797,1717155609198258954,1717155609336265756,1717155609464993206,1717155609591904278,1717155609724916239,1717155609858739558,1717155609992545062,1717155610119162188,1717155610250653809,1717155610388395258,1717155610521258044,1717155610648811221,1717155610781001939,1717155610914565963,1717155611049760362,1717155611191044625,1717155611325152437,1717155611457107054,1717155611589261655,1717155611712361591,1717155611836849526,1717155611962773878,1717155612096684164,1717155612234087575,1717155612379474755,1717155612513064890,1717155612638971614,1717155612769504780,1717155612900623839,1717155613047085023,1717155613179403760,1717155613343167423,1717155613475398460,1717155613595310351,1717155613718553407,1717155613840609667,1717155613961466177,1717155614098830697,1717155614237487135,1717155614383964278,1717155482201513978,1717155482766761638,1717155483490369610,1717155484120016173,1717155484706322943,1717155485240950991,1717155485784191324,1717155486374760961,1717155486901649673,1717155487429241549,1717155487999421050,1717155488719800590,1717155489299061048,1717155489834047973,1717155490408767796,1717155490918623081,1717155491573329406,1717155492103131764,1717155492660051849,1717155493240081995,1717155493913534034,1717155494530808299,1717155495097037293,1717155495687229873,1717155496209020903,1717155496791178396,1717155497313252953,1717155497863441881,1717155498497707645,1717155499079989359,1717155499622684750,1717155500145232634,1717155500710875652,1717155501237146393,1717155501808308371,1717155502383659039,1717155502902638490,1717155503655061238,1717155504209976689,1717155504769103197,1717155505363107954,1717155505888482784,1717155506421631396,1717155506951487220,1717155507493476078,1717155508070738500,1717155508791191432,1717155509308088344,1717155509853290813,1717155510404083347,1717155510871625022,1717155511436892115,1717155511902005298,1717155512442194558,1717155512973771243,1717155513674497012,1717155514216683824,1717155514760328351,1717155515330928533,1717155515914317047,1717155516504950781,1717155517015298604,1717155517622694381],"NumGC":1079745,"NumForcedGC":0,"GCCPUFraction":0.03642135872558363,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438164918,"Frees":438158121},{"Size":16,"Mallocs":6782003385,"Frees":6781642493},{"Size":24,"Mallocs":19178412779,"Frees":19178265368},{"Size":32,"Mallocs":10380390514,"Frees":10380304182},{"Size":48,"Mallocs":1993602958,"Frees":1993555065},{"Size":64,"Mallocs":240874603,"Frees":240842442},{"Size":80,"Mallocs":239114401,"Frees":239099158},{"Size":96,"Mallocs":917537591066,"Frees":917537267018},{"Size":112,"Mallocs":383925073,"Frees":383921105},{"Size":128,"Mallocs":9691307805,"Frees":9691294309},{"Size":144,"Mallocs":190543879,"Frees":190539481},{"Size":160,"Mallocs":71638516,"Frees":71636527},{"Size":176,"Mallocs":5500701,"Frees":5499899},{"Size":192,"Mallocs":32424823,"Frees":32424376},{"Size":208,"Mallocs":68348097,"Frees":68344103},{"Size":224,"Mallocs":3201369,"Frees":3200842},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52442368,"Frees":52414825},{"Size":288,"Mallocs":98698212,"Frees":98676099},{"Size":320,"Mallocs":50009100,"Frees":50005622},{"Size":352,"Mallocs":15156479,"Frees":15155909},{"Size":384,"Mallocs":30486672,"Frees":30486497},{"Size":416,"Mallocs":5934806,"Frees":5932308},{"Size":448,"Mallocs":5902320,"Frees":5898013},{"Size":480,"Mallocs":642394,"Frees":642042},{"Size":512,"Mallocs":39664261,"Frees":39664041},{"Size":576,"Mallocs":21123422,"Frees":21120987},{"Size":640,"Mallocs":1234272,"Frees":1234027},{"Size":704,"Mallocs":18241550,"Frees":18240836},{"Size":768,"Mallocs":6989556,"Frees":6989228},{"Size":896,"Mallocs":7330864,"Frees":7330457},{"Size":1024,"Mallocs":52629012,"Frees":52628839},{"Size":1152,"Mallocs":25002040,"Frees":25001963},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16276822,"Frees":16276793},{"Size":1536,"Mallocs":1802817,"Frees":1802813},{"Size":1792,"Mallocs":12399893,"Frees":12397804},{"Size":2048,"Mallocs":710913,"Frees":710815},{"Size":2304,"Mallocs":22971407,"Frees":22970880},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2039431,"Frees":2039412},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8477538,"Frees":8477527},{"Size":4096,"Mallocs":3541558,"Frees":3541463},{"Size":4864,"Mallocs":8034417,"Frees":8034378},{"Size":5376,"Mallocs":1749899,"Frees":1748355},{"Size":6144,"Mallocs":1312181,"Frees":1312171},{"Size":6528,"Mallocs":6073753,"Frees":6073747},{"Size":6784,"Mallocs":1762772,"Frees":1762700},{"Size":6912,"Mallocs":250168,"Frees":250168},{"Size":8192,"Mallocs":226340,"Frees":226237},{"Size":9472,"Mallocs":1842332,"Frees":1842311},{"Size":9728,"Mallocs":106925,"Frees":106923},{"Size":10240,"Mallocs":1189800,"Frees":1189799},{"Size":10880,"Mallocs":33621,"Frees":33621},{"Size":12288,"Mallocs":273373,"Frees":273369},{"Size":13568,"Mallocs":7836214,"Frees":7836061},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1391264,"Frees":1391233},{"Size":18432,"Mallocs":1193161,"Frees":1193156}]} 2024-05-31T11:40:14.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:40:14.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:40:14.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:40:14.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:40:14.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:40:14.427Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:40:14.502Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:40:14.505Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:40:14.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:40:14.505Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:40:14.506Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:40:15.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:15.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:16.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:16.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:16.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:16.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:17.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:17.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:18.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:18.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:19.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:20.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:20.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:21.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:21.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:22.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:22.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:23.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:23.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:23.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155623377825653, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:40:23.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:23.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:23.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:40:23.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:40:23.391Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=289.724µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:40:24.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:24.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:24.411Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:40:24.411Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:40:24.411Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:40:25.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:25.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:26.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:26.339Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:26.339Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:26.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:27.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:27.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:28.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:28.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:29.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:29.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:40:29.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:40:29.428Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:40:29.500Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:40:30.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:30.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:31.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:31.147Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155631076443370, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:40:31.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:31.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:31.147Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:40:31.148Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:40:31.149Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=677.303µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:40:31.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:32.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:32.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:33.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:33.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:34.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:34.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:40:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:40:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:40:35.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:35.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:36.088Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:36.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:36.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:36.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:37.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:37.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:38.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:38.381Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:39.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:39.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:40.096Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:40.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:41.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:41.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:42.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:42.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:43.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:43.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:44.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:44.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:44.407Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:40:44.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:40:44.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:40:44.410Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:40:44.425Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:40:44.440Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:40:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:40:44.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:40:44.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:40:44.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:40:44.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:40:45.088Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:45.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:46.090Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:46.341Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:46.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:46.381Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:47.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:47.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:48.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:48.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:49.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:49.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:50.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:50.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:51.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:51.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:52.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:52.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:53.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:53.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155653371330191, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:40:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:40:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:40:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:40:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=472.22µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:40:54.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:54.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:40:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:40:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:40:55.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:55.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:56.095Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:56.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:56.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:40:56.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:57.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:58.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:58.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:59.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:40:59.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:40:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:40:59.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:40:59.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:40:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:40:59.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:40:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:40:59.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:40:59.501Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:41:00.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:00.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:01.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155661073651502, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:41:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:41:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:41:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=512.317µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:41:01.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:02.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:02.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:03.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:03.382Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:04.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:04.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:41:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:41:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:41:05.094Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:05.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:06.113Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:06.331Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:06.332Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:06.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:07.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:07.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:08.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:08.393Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:09.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:09.419Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:10.420Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:11.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:12.120Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:12.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:13.146Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:13.403Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:14.156Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:14.408Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:41:14.414Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:41:14.414Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:41:14.435Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:41:14.456Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:41:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:41:14.532Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:41:14.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:15.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:15.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:16.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:16.339Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:16.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:16.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:17.298Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:17.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:18.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:18.595Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:19.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:19.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:20.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:20.438Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:21.308Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:21.459Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:22.295Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:22.528Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:23.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:23.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155682528720276, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:41:23.388Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:23.388Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:23.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:41:23.388Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:41:23.388Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=388.65µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:41:23.662Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:24.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:24.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:41:24.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:41:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:41:24.458Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:25.285Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:25.425Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:26.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:26.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:26.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:26.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:27.285Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:27.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:28.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:28.523Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:29.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:29.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:41:29.437Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:41:29.490Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:41:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:41:29.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:30.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:30.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 225, "num_failedckpts": 0, "time_committing": {"count": 225, "max": 6923, "mean": 76.48444444444445, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155690196293312, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 225, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:41:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:41:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:41:31.135Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=494.873µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:41:31.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:31.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:32.144Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:32.574Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:33.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:33.575Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:33.820Z ERRO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Pipeline has not been started (or has been disabled) - checkpointing is skipped 2024-05-31T11:41:34.336Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:34.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:41:34.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:41:34.409Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:41:34.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:35.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:35.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:35.931Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Done checkpointing with vb list [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511] totalSize=0 2024-05-31T11:41:36.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:36.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:36.348Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:36.584Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:37.018Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Done checkpointing for replication 2024-05-31T11:41:37.018Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt.disableCkptAndWait took 8m42.306481546s 2024-05-31T11:41:37.040Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:37.073Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:37.073Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:41:37.100Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:37.137Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:37.306Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:37.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:37.415Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:37.415Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:41:37.864Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:38.130Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:38.167Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:38.438Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:38.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:39.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:39.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:40.322Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:40.623Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:41.397Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:41.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:42.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:42.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:43.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:43.686Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:44.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:44.450Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:41:44.451Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:41:44.451Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:41:44.451Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:41:44.518Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:41:44.521Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:41:44.592Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:41:44.635Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:45.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:45.531Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:46.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:46.355Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:46.355Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:46.597Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:47.317Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:47.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:48.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:48.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:49.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:49.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:50.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:50.677Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:50.941Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:51.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:51.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:51.777Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:41:51.777Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:41:51.939Z WARN GOXDCR.Utils: ckmgr.mergeAndPersistBrokenMappingDocsAndCkpts took 13.821351346s 2024-05-31T11:41:52.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:52.628Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:52.856Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt took 8m58.145381612s 2024-05-31T11:41:53.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:53.416Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155712629208860, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:41:53.417Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:53.417Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:41:53.417Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:41:53.417Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:41:53.417Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=368.56µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:41:53.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:54.233Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:54.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:41:54.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:41:54.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:41:54.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:55.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:55.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:56.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:56.367Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:56.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:41:56.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:57.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:57.590Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:58.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:58.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:41:59.178Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:41:59.419Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:41:59.461Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:41:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:41:59.497Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:41:59.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:00.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:00.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:01.138Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155720183132789, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:42:01.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:01.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:01.139Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:42:01.139Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:42:01.139Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=488.005µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:42:01.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:01.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:02.300Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:02.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:03.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:03.437Z INFO GOXDCR.AdminPort: doPostPeerToPeerRequest of type PeriodicPush from 10.0.9.103:8091 2024-05-31T11:42:03.437Z ERRO GOXDCR.P2PManager: PeriodicPushHandler Unable to find opaque 903806976 2024-05-31T11:42:03.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:04.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:04.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:42:04.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:42:04.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:42:04.588Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:05.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:05.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:06.332Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:06.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:06.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:06.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:07.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:07.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:08.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:08.639Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:09.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:09.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:10.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:10.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:11.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:11.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:12.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:12.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:13.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:13.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:14.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:14.402Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":154890312,"TotalAlloc":91317097175096,"Sys":327718216,"Lookups":0,"Mallocs":972738712336,"Frees":972737190705,"HeapAlloc":154890312,"HeapSys":308379648,"HeapIdle":130744320,"HeapInuse":177635328,"HeapReleased":68952064,"HeapObjects":1521631,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":2958400,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193930,"GCSys":7462208,"OtherSys":1609294,"NextGC":207360664,"LastGC":1717155734363120843,"PauseTotalNs":3791501107278,"PauseNs":[156173,118193,139711,129987,138331,125828,129132,130240,127797,293911,317373,131520,120370,126216,108131,113672,156092,128981,128888,125350,115004,121460,133189,131774,108530,133053,132919,125501,124004,131260,139104,162113,112936,284457,109565,127783,119886,134912,139339,158613,141093,132749,111137,123016,142275,132688,139296,149668,162139,142986,134052,131367,117738,134951,98869,150601,123212,132140,115337,332972,134637,133613,122332,148858,145892,139928,148698,132527,94816,148829,151603,128848,133504,178455,184621,137067,133684,133604,130798,293966,143187,129457,130822,128155,126604,133117,123189,164465,4641829,137237,137485,136099,115981,6991852,139522,4071546,3513743,118743,114521,210951,140485,122457,132310,9356983,611921,128347,125138,140529,126336,141873,2397059,11840460,141262,311877,131030,4063409,7199103,8367788,12288026,12114633,22930551,12236559,20280493,9104349,20138356,4056070,13385820,8120033,11623321,23293123,23137871,21707327,145436,6959099,183867,15955401,3939226,345250,122384,4101305,11724319,8123159,15791761,17656512,16958909,20684441,178644,12925898,11692116,20810960,7972385,8078704,6755862,17384077,94275,11988783,14856763,744293,4888599,22634048,12264635,14665412,16802047,24079663,2967258,15664222,16130410,21227152,7475121,12409269,13656548,21130538,15419733,12008828,12923387,132608,13299064,10238392,31586731,17884594,8305834,12607940,9625799,29579622,7357638,15534100,16244436,12178358,8048434,7689529,28378602,11424928,23280745,97244,1013450,24358808,137590,8231801,132078,6396904,24144986,19689826,17021620,935110,28310070,12673946,129700,14141855,162663,38664908,8628253,9939132,198991,16748575,12843642,128132,133611,111142,131048,112656,138399,132695,119478,120920,287195,144179,152326,113806,128412,141974,136074,125470,148329,121144,205375,135120,134040,116107,192010,129490,116290,129590,194063,153472,131276,122237,132513,129187,139776,127961,136692,140221,130516,137172,149284,202204],"PauseEnd":[1717155656232322632,1717155656376251116,1717155656505411656,1717155656632480536,1717155656765812472,1717155656896579697,1717155657019554262,1717155657141741729,1717155657274439684,1717155657400443042,1717155657534413843,1717155657667845612,1717155657800021829,1717155657924957825,1717155658065467589,1717155658197451114,1717155658335427269,1717155658469281977,1717155658601423543,1717155658734534338,1717155658868294066,1717155659000106860,1717155659130974457,1717155659256264182,1717155659393349420,1717155659530375066,1717155659662516506,1717155659788058026,1717155659917817811,1717155660065020055,1717155660194339400,1717155660322255373,1717155660447038360,1717155660576649298,1717155660703998209,1717155660831885663,1717155660964328131,1717155661102331080,1717155661233247463,1717155661378948683,1717155661516576511,1717155661643200569,1717155661766243480,1717155661893644315,1717155662020214672,1717155662141921178,1717155662270183724,1717155662399043290,1717155662534557645,1717155662666991473,1717155662798690803,1717155662920236533,1717155663060703450,1717155663192677175,1717155663345944059,1717155663488058174,1717155663619452261,1717155663755463582,1717155663887590100,1717155664020073498,1717155664167295803,1717155664314240695,1717155664460054314,1717155664613041273,1717155664765068222,1717155664905390204,1717155665062107743,1717155665200104533,1717155665360183122,1717155665509953412,1717155665638508721,1717155665774784306,1717155665931669789,1717155666100590036,1717155666253953687,1717155666410499721,1717155666555703148,1717155666707148190,1717155666870125455,1717155667021212274,1717155667173054053,1717155667308761372,1717155667444417573,1717155667577532390,1717155667722562171,1717155667859372964,1717155668006839275,1717155668148496098,1717155668347440834,1717155668520779127,1717155668655722376,1717155668793160782,1717155668938364506,1717155669147547802,1717155669365773410,1717155669587884388,1717155669783117371,1717155669989327619,1717155670202666791,1717155670423187970,1717155670602352509,1717155670816921327,1717155671005606865,1717155671238742461,1717155671443703191,1717155671666992362,1717155671871873167,1717155672072657925,1717155672277696787,1717155672487668485,1717155672675030723,1717155672885143563,1717155673088158289,1717155673296233212,1717155673556367439,1717155673768814738,1717155673962918360,1717155674399093944,1717155674944515073,1717155675519769708,1717155676054908356,1717155676650653141,1717155677268859602,1717155677831200505,1717155678434455457,1717155679016327958,1717155679613403638,1717155680154254491,1717155680719561447,1717155681244412181,1717155681785378872,1717155682330033591,1717155682887182883,1717155683487654465,1717155684100571766,1717155684665260216,1717155685205928656,1717155685735124256,1717155686263885676,1717155686924839098,1717155687581993535,1717155688118566072,1717155688735660018,1717155689316386761,1717155689865168376,1717155690405912712,1717155690898975710,1717155691451784924,1717155691932467747,1717155692493342937,1717155692998179484,1717155693683178730,1717155694333401193,1717155694896471202,1717155695548264094,1717155696092242724,1717155696787323516,1717155697503251201,1717155698219462514,1717155699056293181,1717155699777297032,1717155700605283546,1717155701344363254,1717155702030580379,1717155702739167897,1717155703633736125,1717155704482558858,1717155705178405858,1717155705918389915,1717155706675167145,1717155707478533798,1717155708301542391,1717155709055509924,1717155709775490830,1717155710644727900,1717155711423286854,1717155712045005248,1717155712724543785,1717155713384190098,1717155714131399172,1717155714703035165,1717155715253712817,1717155715784086451,1717155716410417170,1717155716884115758,1717155717517520879,1717155718088416249,1717155718792339205,1717155719371467869,1717155719868195005,1717155720404857748,1717155720982228300,1717155721667992598,1717155722229805706,1717155722771530116,1717155723324412833,1717155723966488498,1717155724571363612,1717155725050720774,1717155725656083145,1717155726278643797,1717155726844300486,1717155727387007632,1717155727954989492,1717155728637882974,1717155729218994821,1717155729754948214,1717155730322337541,1717155730859277216,1717155731508652318,1717155731982496689,1717155732533694062,1717155733090162917,1717155733797953689,1717155734363120843,1717155650802133216,1717155650935670377,1717155651085034371,1717155651225642189,1717155651385134309,1717155651522989635,1717155651642512621,1717155651768826891,1717155651895446085,1717155652025144717,1717155652148275614,1717155652283332204,1717155652410325065,1717155652532233038,1717155652660984953,1717155652793637958,1717155652918195400,1717155653060181467,1717155653183761479,1717155653346977515,1717155653483399825,1717155653617082331,1717155653752609087,1717155653884311174,1717155654017521762,1717155654144550852,1717155654270422450,1717155654397069455,1717155654528018563,1717155654648326577,1717155654775223634,1717155654901226514,1717155655026391556,1717155655147803556,1717155655266966843,1717155655389773237,1717155655520821810,1717155655654014406,1717155655804963926,1717155655949008342,1717155656098262457],"NumGC":1080279,"NumForcedGC":0,"GCCPUFraction":0.03643061745572611,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438316212,"Frees":438309863},{"Size":16,"Mallocs":6784562466,"Frees":6784202737},{"Size":24,"Mallocs":19183701045,"Frees":19183564035},{"Size":32,"Mallocs":10383254844,"Frees":10383177178},{"Size":48,"Mallocs":1994300623,"Frees":1994254365},{"Size":64,"Mallocs":240941766,"Frees":240909649},{"Size":80,"Mallocs":239168281,"Frees":239153062},{"Size":96,"Mallocs":918032138943,"Frees":918031404016},{"Size":112,"Mallocs":384016291,"Frees":384005483},{"Size":128,"Mallocs":9693920358,"Frees":9693904096},{"Size":144,"Mallocs":190618832,"Frees":190614438},{"Size":160,"Mallocs":71651073,"Frees":71648364},{"Size":176,"Mallocs":5502339,"Frees":5501537},{"Size":192,"Mallocs":32433770,"Frees":32433323},{"Size":208,"Mallocs":68364109,"Frees":68360059},{"Size":224,"Mallocs":3201720,"Frees":3201193},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52451558,"Frees":52424018},{"Size":288,"Mallocs":98768830,"Frees":98746729},{"Size":320,"Mallocs":50015680,"Frees":50011841},{"Size":352,"Mallocs":15164601,"Frees":15164033},{"Size":384,"Mallocs":30495354,"Frees":30495146},{"Size":416,"Mallocs":5936157,"Frees":5933663},{"Size":448,"Mallocs":5902347,"Frees":5898040},{"Size":480,"Mallocs":642396,"Frees":642044},{"Size":512,"Mallocs":39676247,"Frees":39675979},{"Size":576,"Mallocs":21139109,"Frees":21136678},{"Size":640,"Mallocs":1234528,"Frees":1234282},{"Size":704,"Mallocs":18247008,"Frees":18245934},{"Size":768,"Mallocs":6991604,"Frees":6991270},{"Size":896,"Mallocs":7332886,"Frees":7332474},{"Size":1024,"Mallocs":52642625,"Frees":52642382},{"Size":1152,"Mallocs":25018961,"Frees":25018885},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16281229,"Frees":16280840},{"Size":1536,"Mallocs":1803308,"Frees":1803304},{"Size":1792,"Mallocs":12407610,"Frees":12405515},{"Size":2048,"Mallocs":711160,"Frees":711063},{"Size":2304,"Mallocs":22989739,"Frees":22989215},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2039821,"Frees":2039802},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8480296,"Frees":8480280},{"Size":4096,"Mallocs":3542318,"Frees":3542224},{"Size":4864,"Mallocs":8038743,"Frees":8038706},{"Size":5376,"Mallocs":1750479,"Frees":1748935},{"Size":6144,"Mallocs":1313741,"Frees":1313731},{"Size":6528,"Mallocs":6075798,"Frees":6075788},{"Size":6784,"Mallocs":1763485,"Frees":1763412},{"Size":6912,"Mallocs":250228,"Frees":250228},{"Size":8192,"Mallocs":226394,"Frees":226292},{"Size":9472,"Mallocs":1842950,"Frees":1842927},{"Size":9728,"Mallocs":106972,"Frees":106972},{"Size":10240,"Mallocs":1190040,"Frees":1190039},{"Size":10880,"Mallocs":33633,"Frees":33633},{"Size":12288,"Mallocs":273442,"Frees":273438},{"Size":13568,"Mallocs":7838972,"Frees":7838805},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1391658,"Frees":1391628},{"Size":18432,"Mallocs":1193590,"Frees":1193581}]} 2024-05-31T11:42:14.418Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:42:14.418Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:42:14.419Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:42:14.420Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:42:14.508Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:42:14.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:14.539Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:42:14.560Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:42:15.152Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:15.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:16.196Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:16.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:16.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:16.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:17.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:17.444Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:18.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:18.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:19.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:19.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:20.161Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:20.553Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:21.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:21.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:22.241Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:22.451Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:23.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:23.418Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155742451432451, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:42:23.418Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:23.418Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:23.418Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:42:23.418Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:42:23.419Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=414.451µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:42:23.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:24.156Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:24.422Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:42:24.422Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:42:24.422Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:42:24.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:25.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:25.622Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:26.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:26.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:26.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:26.459Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:27.294Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:27.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:28.258Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:28.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:29.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:29.405Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:42:29.513Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:42:29.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:29.522Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:42:29.596Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:42:30.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:30.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:31.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155750215336358, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:42:31.167Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:31.182Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:31.182Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:42:31.182Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:42:31.182Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=474.034µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:42:31.265Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:31.444Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:32.265Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:32.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:33.253Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:33.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:34.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:34.407Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:42:34.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:42:34.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:42:34.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:35.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:35.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:36.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:36.409Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:36.411Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:36.594Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:37.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:37.448Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:38.298Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:38.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:39.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:39.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:40.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:40.464Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:41.295Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:41.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:42.307Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:42.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:43.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:43.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:44.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:44.432Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:42:44.432Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:42:44.432Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:42:44.432Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:42:44.520Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:42:44.521Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:42:44.590Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:42:44.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:45.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:45.467Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:46.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:46.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:46.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:46.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:47.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:47.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:48.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:48.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:49.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:49.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:50.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:50.460Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:51.351Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:51.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:52.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:52.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:53.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:53.395Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 367, "num_failedckpts": 0, "time_committing": {"count": 367, "max": 1342, "mean": 26.997275204359674, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155772518533645, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 367, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:42:53.395Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:53.396Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:42:53.396Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:42:53.396Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:42:53.396Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=430.898µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:42:53.709Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:54.187Z ERRO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Pipeline has not been started (or has been disabled) - checkpointing is skipped 2024-05-31T11:42:54.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:54.401Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:42:54.401Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:42:54.401Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:42:54.497Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:55.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:55.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:56.065Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Done checkpointing with vb list [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511] totalSize=0 2024-05-31T11:42:56.233Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:56.439Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:56.458Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:42:56.591Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:57.342Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Done checkpointing for replication 2024-05-31T11:42:57.342Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt.disableCkptAndWait took 10m2.628768969s 2024-05-31T11:42:57.409Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:57.422Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:57.422Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:42:57.429Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:57.457Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:57.509Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:57.577Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:57.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:57.633Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc backfill_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:57.633Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:42:58.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:58.694Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:58.824Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:58.825Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:42:59.347Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:42:59.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:42:59.502Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:42:59.502Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:42:59.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:42:59.558Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:43:00.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:00.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:01.144Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155780291502532, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:43:01.144Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:01.144Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:01.144Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:43:01.144Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:43:01.144Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=492.71µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:43:01.331Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:01.626Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:02.313Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:02.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:03.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:03.604Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:04.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:04.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:43:04.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:43:04.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:43:04.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:05.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:05.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:06.323Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:06.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:06.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:06.613Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:07.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:07.734Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:08.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:08.730Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:09.347Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:09.636Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:10.332Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:10.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:11.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:11.541Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:12.322Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:12.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:13.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:13.717Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:14.160Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:43:14.311Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:14.422Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:43:14.422Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:43:14.422Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:43:14.423Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:43:14.556Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:43:14.563Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:43:14.597Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:43:14.672Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:15.420Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:15.485Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 retrieved an existing mappingsDoc 2024-05-31T11:43:15.487Z INFO GOXDCR.CheckpointSvc: Loaded brokenMap: map[] 2024-05-31T11:43:15.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:15.677Z WARN GOXDCR.Utils: ckmgr.mergeAndPersistBrokenMappingDocsAndCkpts took 16.865803725s 2024-05-31T11:43:16.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:16.367Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:16.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:16.600Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:16.996Z WARN GOXDCR.Utils: ckmgr.stopTheWorldAndMergeCkpt took 10m22.28296714s 2024-05-31T11:43:16.996Z INFO GOXDCR.P2PManager: Done handling peer-to-peer push requests from 10.0.9.103:8091 (opaque 1229651968 timeTaken 10m22.287215245s) 2024-05-31T11:43:17.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:17.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:18.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:18.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:19.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:19.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:20.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:20.541Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:21.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:21.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:22.258Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:22.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:23.327Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:23.389Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155802492209601, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:43:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:43:23.390Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:43:23.390Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=385.664µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:43:23.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:24.233Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:24.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:43:24.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:43:24.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:43:24.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:25.279Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:25.469Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:26.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:26.364Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:26.365Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:26.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:27.271Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:27.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:28.339Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:28.574Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:29.283Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:43:29.447Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:43:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:43:29.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:29.505Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:43:30.332Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:30.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:31.138Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155810355489388, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:43:31.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:31.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:31.138Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:43:31.138Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:43:31.139Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=472.092µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:43:31.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:31.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:32.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:32.641Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:33.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:33.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:34.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:34.406Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:43:34.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:43:34.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:43:34.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:35.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:35.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:36.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:36.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:36.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:37.249Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:37.335Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:37.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:38.233Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:38.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:39.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:39.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:40.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:40.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:41.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:41.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:42.311Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:42.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:43.138Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:43.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:44.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:43:44.417Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:43:44.419Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:43:44.419Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:43:44.458Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:43:44.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:44.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:43:44.526Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:43:45.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:45.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:46.148Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:46.358Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:46.374Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:46.589Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:47.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:47.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:48.300Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:48.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:49.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:49.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:50.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:50.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:51.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:51.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:52.286Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:52.450Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:53.295Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:53.387Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155832451040445, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:43:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:53.387Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:43:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:43:53.387Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:43:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=377.516µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:43:53.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:54.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:54.407Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:43:54.407Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:43:54.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:43:54.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:55.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:55.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:56.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:56.352Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:56.352Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:43:56.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:57.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:57.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:58.210Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:58.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:59.206Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:43:59.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:43:59.435Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:43:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:43:59.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:43:59.522Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:44:00.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:00.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:01.146Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155840192701863, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:44:01.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:01.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:01.147Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:44:01.147Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:44:01.147Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=482.67µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:44:01.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:01.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:02.346Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:02.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:03.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:03.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:04.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:04.448Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:44:04.449Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:44:04.449Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:44:04.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:05.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:05.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:06.317Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:06.345Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:06.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:06.521Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:07.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:07.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:08.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:08.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:09.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:09.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:10.171Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:10.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:11.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:11.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:12.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:12.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:13.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:13.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:14.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:14.425Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":191906560,"TotalAlloc":91337241273880,"Sys":327718216,"Lookups":0,"Mallocs":972954748365,"Frees":972952788359,"HeapAlloc":191906560,"HeapSys":308412416,"HeapIdle":93847552,"HeapInuse":214564864,"HeapReleased":65691648,"HeapObjects":1960006,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3658560,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193930,"GCSys":7429440,"OtherSys":1609294,"NextGC":214406368,"LastGC":1717155853950393051,"PauseTotalNs":3793704850246,"PauseNs":[14705299,138460,21677103,12518581,25013791,8416240,28042833,12134130,16562631,14260069,12831237,11728415,14466443,135600,131849,19946965,8357527,13045036,7658355,14742936,9311031,148893,21468761,10349478,4213560,8525382,6745403,141579,18090020,19459738,4038539,15291068,23672066,9375464,10034190,13061973,8332751,28895342,21253965,23213641,16056413,2949182,8469359,18225423,9853609,25609192,3912104,3978247,102857,18186793,16320846,22170392,106408,19786490,13790600,10569310,13439594,1820197,7653275,10243028,7258772,2194179,16604323,16356084,133048,11848389,11765388,144604,292507,11969893,18984072,5502742,15402871,24595214,12366407,9499465,19730468,19755890,12144311,19627666,10163559,94993,138573,7902562,4029895,16536746,22754968,48702795,16673427,20926718,11729121,16143307,9399164,20449161,117791,12739983,7329565,298755,15725798,154289,13289476,128666,20280294,26018836,3525148,12805637,28439673,140371,12063520,144084,11304358,8187489,17190207,186420,11894099,9914363,8112914,3653810,89879,21189931,17048843,9634310,8321221,18711170,13724697,11796947,12158861,11991221,11121994,20532463,14549294,8080904,3197004,23242818,8781710,3942776,26124784,13829605,7250363,154901,399235,4117460,302701,9869456,27420990,130625,12280682,20457158,249493,22596429,4125476,3134677,132333,7190797,4031482,8547746,14856763,744293,4888599,22634048,12264635,14665412,16802047,24079663,2967258,15664222,16130410,21227152,7475121,12409269,13656548,21130538,15419733,12008828,12923387,132608,13299064,10238392,31586731,17884594,8305834,12607940,9625799,29579622,7357638,15534100,16244436,12178358,8048434,7689529,28378602,11424928,23280745,97244,1013450,24358808,137590,8231801,132078,6396904,24144986,19689826,17021620,935110,28310070,12673946,129700,14141855,162663,38664908,8628253,9939132,198991,16748575,12843642,141651,1098973,1989890,8891803,12058046,4204646,11938675,18167468,15733230,12164040,12124968,16411608,12511408,21387476,11381012,120567,20527591,228576,10842338,12208240,18531296,25533395,532030,19400190,11901656,9613833,10371533,208924,142519,6859238,8091158,3421685,129665,25446327,18275983,136219,4044055,18311270,2892397,2638691,21819771],"PauseEnd":[1717155757731630272,1717155758230824962,1717155758903069433,1717155759490544105,1717155760033585532,1717155760611356648,1717155761101531780,1717155761680540053,1717155762252005270,1717155762766812377,1717155763313843733,1717155763989918350,1717155764637550176,1717155765153265577,1717155765651091321,1717155766167155479,1717155766718514844,1717155767254504319,1717155767820829461,1717155768400387899,1717155769020128866,1717155769607903767,1717155770110738741,1717155770674990242,1717155771207992261,1717155771791220089,1717155772371349462,1717155772882206276,1717155773576439117,1717155774280162874,1717155774827401218,1717155775402123355,1717155775900448648,1717155776472293307,1717155777062936452,1717155777822683847,1717155778494932827,1717155779279238956,1717155780047890890,1717155780790655106,1717155781580456806,1717155782330702245,1717155783048241735,1717155783850802943,1717155784637926511,1717155785433257721,1717155786151171538,1717155786880812512,1717155787818974612,1717155788644987422,1717155789564240210,1717155790419280809,1717155791277826784,1717155792035615970,1717155792883834080,1717155793803207549,1717155794671324350,1717155795313672758,1717155795985724395,1717155796691420347,1717155797342675985,1717155797973738267,1717155798676977248,1717155799379786455,1717155799887035009,1717155800487617625,1717155800977756904,1717155801611782486,1717155802091371147,1717155802718907424,1717155803409716727,1717155804071394176,1717155804681960005,1717155805174834620,1717155805752546399,1717155806330090585,1717155807003648012,1717155807675008311,1717155808289295393,1717155809138275913,1717155809733363431,1717155810283080993,1717155810837578795,1717155811435628402,1717155811969117218,1717155812563714267,1717155813078544602,1717155813794055342,1717155814493850553,1717155815003621915,1717155815603235028,1717155816120072218,1717155816742049278,1717155817473020648,1717155818011708905,1717155818630993162,1717155819371357856,1717155819911680301,1717155820538127663,1717155821056084461,1717155821689831877,1717155822271702333,1717155822846190565,1717155823436527436,1717155824080848564,1717155824723033879,1717155825349077623,1717155825904121309,1717155826527109538,1717155827003939938,1717155827627927584,1717155828190928184,1717155828862745889,1717155829541721981,1717155830014975646,1717155830629879675,1717155831128307639,1717155831706474643,1717155832228469507,1717155832758635769,1717155833316859535,1717155833960356199,1717155834664290899,1717155835206198066,1717155835764784237,1717155836342574358,1717155836922154925,1717155837497946811,1717155838013288190,1717155838655047610,1717155839329711158,1717155839955819505,1717155840590922014,1717155841114574669,1717155841700333683,1717155842225387605,1717155842798744885,1717155843356559103,1717155844026929106,1717155844624338905,1717155845171813690,1717155845735126269,1717155846270959328,1717155846825981457,1717155847403179747,1717155847928588881,1717155848577083583,1717155849296475789,1717155849838889024,1717155850489058939,1717155851018501560,1717155851638801224,1717155852207435967,1717155852769287142,1717155853301334912,1717155853950393051,1717155696787323516,1717155697503251201,1717155698219462514,1717155699056293181,1717155699777297032,1717155700605283546,1717155701344363254,1717155702030580379,1717155702739167897,1717155703633736125,1717155704482558858,1717155705178405858,1717155705918389915,1717155706675167145,1717155707478533798,1717155708301542391,1717155709055509924,1717155709775490830,1717155710644727900,1717155711423286854,1717155712045005248,1717155712724543785,1717155713384190098,1717155714131399172,1717155714703035165,1717155715253712817,1717155715784086451,1717155716410417170,1717155716884115758,1717155717517520879,1717155718088416249,1717155718792339205,1717155719371467869,1717155719868195005,1717155720404857748,1717155720982228300,1717155721667992598,1717155722229805706,1717155722771530116,1717155723324412833,1717155723966488498,1717155724571363612,1717155725050720774,1717155725656083145,1717155726278643797,1717155726844300486,1717155727387007632,1717155727954989492,1717155728637882974,1717155729218994821,1717155729754948214,1717155730322337541,1717155730859277216,1717155731508652318,1717155731982496689,1717155732533694062,1717155733090162917,1717155733797953689,1717155734363120843,1717155734900961794,1717155735427921713,1717155735914603519,1717155736490697173,1717155737008486008,1717155737640092920,1717155738190795266,1717155738850525263,1717155739389168616,1717155739912103658,1717155740463569627,1717155740965414040,1717155741591700014,1717155742098056237,1717155742641829095,1717155743164594950,1717155743864437737,1717155744471525687,1717155744972766064,1717155745596508583,1717155746195579222,1717155746754852374,1717155747309895614,1717155747845108010,1717155748412477635,1717155748990650382,1717155749583191255,1717155750061310364,1717155750616162093,1717155751057702310,1717155751656603573,1717155752175152746,1717155752679134718,1717155753176555553,1717155753836730855,1717155754428147710,1717155754962134285,1717155755581352159,1717155756003154259,1717155756545350131,1717155757123961270],"NumGC":1080476,"NumForcedGC":0,"GCCPUFraction":0.036442591651277535,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438478393,"Frees":438471762},{"Size":16,"Mallocs":6787346922,"Frees":6786985626},{"Size":24,"Mallocs":19189002718,"Frees":19188842631},{"Size":32,"Mallocs":10386175909,"Frees":10386083415},{"Size":48,"Mallocs":1995051104,"Frees":1995001951},{"Size":64,"Mallocs":241008695,"Frees":240978081},{"Size":80,"Mallocs":239226861,"Frees":239211615},{"Size":96,"Mallocs":918231377924,"Frees":918230237311},{"Size":112,"Mallocs":384107504,"Frees":384103537},{"Size":128,"Mallocs":9696545641,"Frees":9696525364},{"Size":144,"Mallocs":190704497,"Frees":190700112},{"Size":160,"Mallocs":71663604,"Frees":71661616},{"Size":176,"Mallocs":5503935,"Frees":5503133},{"Size":192,"Mallocs":32442397,"Frees":32441953},{"Size":208,"Mallocs":68379702,"Frees":68375710},{"Size":224,"Mallocs":3202064,"Frees":3201538},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52460741,"Frees":52433186},{"Size":288,"Mallocs":98865606,"Frees":98847089},{"Size":320,"Mallocs":50022261,"Frees":50018797},{"Size":352,"Mallocs":15172251,"Frees":15171710},{"Size":384,"Mallocs":30503707,"Frees":30503533},{"Size":416,"Mallocs":5937519,"Frees":5935041},{"Size":448,"Mallocs":5902374,"Frees":5898067},{"Size":480,"Mallocs":642396,"Frees":642044},{"Size":512,"Mallocs":39688115,"Frees":39687869},{"Size":576,"Mallocs":21160940,"Frees":21158507},{"Size":640,"Mallocs":1234788,"Frees":1234543},{"Size":704,"Mallocs":18252456,"Frees":18251742},{"Size":768,"Mallocs":6993324,"Frees":6992997},{"Size":896,"Mallocs":7334796,"Frees":7334392},{"Size":1024,"Mallocs":52656280,"Frees":52656071},{"Size":1152,"Mallocs":25042034,"Frees":25041960},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16285633,"Frees":16285604},{"Size":1536,"Mallocs":1803806,"Frees":1803802},{"Size":1792,"Mallocs":12411807,"Frees":12409722},{"Size":2048,"Mallocs":711335,"Frees":711238},{"Size":2304,"Mallocs":23012234,"Frees":23011710},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2040215,"Frees":2040198},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8482596,"Frees":8482587},{"Size":4096,"Mallocs":3543083,"Frees":3542994},{"Size":4864,"Mallocs":8046645,"Frees":8046609},{"Size":5376,"Mallocs":1751571,"Frees":1750027},{"Size":6144,"Mallocs":1316837,"Frees":1316827},{"Size":6528,"Mallocs":6077518,"Frees":6077513},{"Size":6784,"Mallocs":1764065,"Frees":1763994},{"Size":6912,"Mallocs":250288,"Frees":250288},{"Size":8192,"Mallocs":226447,"Frees":226345},{"Size":9472,"Mallocs":1843562,"Frees":1843539},{"Size":9728,"Mallocs":107019,"Frees":107019},{"Size":10240,"Mallocs":1190279,"Frees":1190278},{"Size":10880,"Mallocs":33645,"Frees":33645},{"Size":12288,"Mallocs":273511,"Frees":273507},{"Size":13568,"Mallocs":7841272,"Frees":7841122},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1392052,"Frees":1392022},{"Size":18432,"Mallocs":1194018,"Frees":1194010}]} 2024-05-31T11:44:14.425Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:44:14.425Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:44:14.425Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:44:14.425Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:44:14.487Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:44:14.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:44:14.542Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:44:14.542Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:44:14.542Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:44:14.558Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:44:14.605Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:44:14.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:15.262Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:15.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:16.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:16.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:16.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:16.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:17.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:17.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:18.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:18.583Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:19.324Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:19.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:20.351Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:20.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:21.165Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:21.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:22.170Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:22.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:23.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:23.390Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155862582235404, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:44:23.390Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:23.390Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:23.390Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:44:23.390Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:44:23.391Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=470.959µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:44:23.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:24.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:24.415Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:44:24.415Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:44:24.415Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:44:24.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:25.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:25.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:26.372Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:26.496Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:26.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:26.730Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:27.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:27.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:28.176Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:28.606Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:29.307Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:44:29.460Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:44:29.494Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:44:29.498Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:44:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:44:29.498Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:44:29.519Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:44:29.520Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:44:29.538Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:30.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:30.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155870310515490, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:44:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:44:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:44:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=463.98µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:44:31.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:31.679Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:32.252Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:32.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:33.303Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:33.465Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:34.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:34.418Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:44:34.418Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:44:34.418Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:44:34.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:35.155Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:35.609Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:36.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:36.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:36.361Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:36.453Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:37.281Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:37.453Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:38.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:38.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:39.221Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:39.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:40.249Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:40.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:41.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:41.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:42.261Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:42.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:43.298Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:43.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:44.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:44.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:44:44.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:44:44.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:44:44.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:44:44.448Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:44:44.512Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:44:44.519Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:44:44.539Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:44:44.539Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:44:44.539Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:44:44.540Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:44:44.597Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:45.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:45.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:46.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:46.376Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:46.377Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:46.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:47.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:47.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:48.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:48.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:49.320Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:49.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:50.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:50.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:51.160Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:51.594Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:52.207Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:52.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:53.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155892592756131, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:44:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:44:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:44:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:44:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=381.94µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:44:53.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:54.213Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:54.403Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:44:54.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:44:54.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:44:54.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:55.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:55.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:56.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:56.377Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:56.394Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:44:56.579Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:57.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:57.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:58.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:58.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:44:59.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:44:59.413Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:44:59.481Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:44:59.500Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:44:59.501Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:44:59.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:00.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:00.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:01.140Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155900173134277, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:45:01.140Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:01.140Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:01.141Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:45:01.141Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:45:01.141Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=493.003µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:45:01.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:01.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:02.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:02.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:03.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:03.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:04.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:04.423Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:45:04.423Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:45:04.423Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:45:04.575Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:05.171Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:05.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:06.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:06.379Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:06.379Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:06.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:07.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:07.531Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:08.302Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:08.482Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:10.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:10.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:11.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:11.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:12.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:12.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:13.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:13.369Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:14.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:14.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:14.406Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:45:14.407Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:45:14.407Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:45:14.407Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:45:14.423Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:45:14.438Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:45:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:45:15.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:15.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:16.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:16.325Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:16.326Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:16.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:17.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:17.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:18.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:18.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:19.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:19.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:20.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:20.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:21.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:21.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:22.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:22.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:23.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:23.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:23.399Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155923361216876, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:45:23.399Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:23.399Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:23.399Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:45:23.399Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:45:23.400Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=412.896µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:45:24.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:24.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:24.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:45:24.409Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:45:24.409Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:45:25.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:25.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:26.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:26.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:26.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:26.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:27.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:28.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:28.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:29.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:29.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:29.401Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:45:29.417Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:45:29.439Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:45:29.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:45:29.506Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:45:29.506Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:45:29.506Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:45:29.507Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:45:30.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:30.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:31.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155931071182343, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:45:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:45:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:45:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=495.675µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:45:31.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:32.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:32.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:33.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:33.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:34.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:34.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:34.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:45:34.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:45:34.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:45:35.074Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:35.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:36.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:36.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:36.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:37.089Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:37.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:38.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:38.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:39.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:39.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:40.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:40.375Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:41.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:41.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:42.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:42.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:43.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:43.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:44.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:44.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:45:44.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:45:44.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:45:44.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:45:44.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:45:44.424Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:45:44.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:45:45.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:45.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:46.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:46.346Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:46.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:46.375Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:47.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:47.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:48.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:48.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:49.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:49.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:50.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:50.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:51.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:51.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:52.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:52.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:53.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:53.373Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:53.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155953374923299, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:45:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:53.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:45:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:45:53.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:45:53.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=228.795µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:45:54.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:54.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:45:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:45:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:45:55.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:55.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:56.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:56.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:56.351Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:45:56.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:57.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:57.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:58.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:58.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:59.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:45:59.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:45:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:45:59.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:45:59.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:45:59.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:45:59.500Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:45:59.500Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:45:59.500Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:45:59.501Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:00.093Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:00.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:01.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155961086194444, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:46:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:46:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:46:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=474.641µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:46:01.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:02.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:02.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:03.068Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:03.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:04.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:04.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:04.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:46:04.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:46:04.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:46:05.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:05.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:06.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:06.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:06.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:06.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:07.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:07.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:08.104Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:08.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:09.094Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:09.442Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:10.085Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:10.411Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:11.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:11.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:12.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:12.398Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:13.142Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:13.398Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:14.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:14.404Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":191660192,"TotalAlloc":91388622292400,"Sys":327718216,"Lookups":0,"Mallocs":973496256372,"Frees":973494309560,"HeapAlloc":191660192,"HeapSys":308379648,"HeapIdle":93618176,"HeapInuse":214761472,"HeapReleased":83066880,"HeapObjects":1946812,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3661120,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3193930,"GCSys":7462208,"OtherSys":1609294,"NextGC":211143080,"LastGC":1717155973986292611,"PauseTotalNs":3794890376380,"PauseNs":[125721,127047,108966,127639,132067,135428,133043,133738,107557,109800,133836,149941,141125,138680,128919,233624,135838,119410,120169,109594,165218,129107,138799,127982,128693,125447,112417,137566,129725,129400,133063,124004,129616,107989,120135,114955,114469,292399,135710,133663,119874,124106,126508,131521,135133,338984,126395,146432,109304,134818,110397,287736,127860,117076,137281,333477,117095,291377,115924,125243,165427,118413,133647,127145,138393,130984,137511,121272,142615,155229,149317,135113,121446,124966,134385,124585,128960,122315,131469,134083,124009,122602,108158,124370,142083,126437,295246,122786,140677,278860,125443,135639,152645,142342,120227,155789,110772,142521,123541,286107,118738,207997,128382,121752,131979,191639,128852,123193,113043,103272,128766,139053,125937,129592,122996,145089,145355,124133,145741,130076,130355,139532,133948,123622,124955,126372,131274,186604,119257,129905,289243,122690,127369,133998,117096,133204,108337,159289,127456,126896,129195,84219,136256,285882,187878,122716,128949,170243,126591,131174,129806,133379,168566,129818,129613,174422,121475,130643,131405,136009,111438,131469,202648,143247,138524,121635,126839,130626,131245,133408,302705,131702,253027,126283,126711,118495,130990,131465,140529,179339,126952,118027,134797,128274,133489,128077,139975,189243,121792,142415,136082,125520,110951,288764,129988,122353,163011,108089,117491,114662,6935636,112435,133281,128536,129910,130569,121271,131475,127148,132119,300390,150833,127991,120640,5721588,7372779,293076,138439,124985,121857,7736014,130970,147882,106818,411158,146680,302879,117219,4962124,294743,286909,298665,141870,132795,143337,143482,125709,11331563,171342,215235,135420,119558,109618,111808,298717,136331,152448,129388,141801,141998,118731,118189,121759,126995,142707,111465],"PauseEnd":[1717155941759146022,1717155941880487142,1717155942001200959,1717155942123468759,1717155942252032752,1717155942387685030,1717155942514960212,1717155942638560200,1717155942760266839,1717155942882610857,1717155943006153516,1717155943119275050,1717155943242884395,1717155943376649783,1717155943494462662,1717155943646059087,1717155943782881079,1717155943899236044,1717155944020408700,1717155944138640521,1717155944261467992,1717155944388305492,1717155944517165236,1717155944645958059,1717155944772597083,1717155944893201509,1717155945015691727,1717155945133855346,1717155945255830816,1717155945388022819,1717155945512922568,1717155945635190117,1717155945763556783,1717155945885630179,1717155946001158260,1717155946117999066,1717155946238791956,1717155946374483256,1717155946503774501,1717155946624412910,1717155946760647777,1717155946881277154,1717155947003120943,1717155947126322976,1717155947248645150,1717155947380065590,1717155947506444353,1717155947626209976,1717155947745201882,1717155947864617411,1717155947988222875,1717155948110056545,1717155948238604131,1717155948376171144,1717155948506766314,1717155948646129385,1717155948773543315,1717155948895555770,1717155949018233468,1717155949134672100,1717155949257310060,1717155949390784020,1717155949517394193,1717155949639848765,1717155949761845509,1717155949883219424,1717155950006118736,1717155950116672290,1717155950238644682,1717155950379507563,1717155950510564514,1717155950622334096,1717155950745591840,1717155950876121223,1717155951004641554,1717155951122978496,1717155951242494858,1717155951375690070,1717155951506375892,1717155951621317250,1717155951744528196,1717155951871807239,1717155951992862735,1717155952110226382,1717155952237365769,1717155952373225956,1717155952498357245,1717155952624892891,1717155952748399278,1717155952865112622,1717155952982767439,1717155953103491478,1717155953226871051,1717155953359170667,1717155953487919147,1717155953617785421,1717155953740211869,1717155953862775206,1717155953983737439,1717155954098636990,1717155954225063032,1717155954372539025,1717155954505668482,1717155954628246587,1717155954751865866,1717155954874121267,1717155955002946019,1717155955127227344,1717155955257140823,1717155955383365143,1717155955509115133,1717155955631233185,1717155955753171236,1717155955872723324,1717155955993729090,1717155956111520827,1717155956241529111,1717155956380730088,1717155956510830525,1717155956628873646,1717155956752188118,1717155956881763347,1717155957011616288,1717155957128202721,1717155957252053609,1717155957387906598,1717155957515562781,1717155957634535193,1717155957759447993,1717155957881155587,1717155958004542371,1717155958124409112,1717155958246547853,1717155958379383198,1717155958508364142,1717155958643039149,1717155958756999006,1717155958878031497,1717155958998755769,1717155959111048873,1717155959229189835,1717155959367332987,1717155959499112148,1717155959621227978,1717155959742176683,1717155959860741386,1717155959987527548,1717155960121770755,1717155960240209348,1717155960373049883,1717155960498438072,1717155960619482665,1717155960748910606,1717155960869650415,1717155960990558129,1717155961123868721,1717155961251642306,1717155961385199638,1717155961514342495,1717155961644338229,1717155961777224609,1717155961898122663,1717155962025501538,1717155962150816226,1717155962278493992,1717155962403017643,1717155962526868530,1717155962639269224,1717155962759416457,1717155962882025611,1717155963002497485,1717155963125719951,1717155963244968470,1717155963377158238,1717155963510888986,1717155963658904643,1717155963789264129,1717155963911346613,1717155964031555458,1717155964169225059,1717155964305194916,1717155964443695937,1717155964586127787,1717155964729217973,1717155964864687304,1717155965018022849,1717155965163192127,1717155965312352206,1717155965466480412,1717155965606955995,1717155965742550957,1717155965884068809,1717155966023562616,1717155966164349037,1717155966302155701,1717155966441379465,1717155966571493990,1717155966709719659,1717155966847655675,1717155966972857271,1717155967121408502,1717155967255884493,1717155967409428498,1717155967541502879,1717155967670711791,1717155967799436849,1717155967935561326,1717155968069424649,1717155968214354242,1717155968369737636,1717155968522676035,1717155968689284182,1717155968841417381,1717155968980781031,1717155969180070585,1717155969399222048,1717155969596146809,1717155969780876091,1717155969969508795,1717155970167911464,1717155970383124785,1717155970580572259,1717155970764645197,1717155970936905209,1717155971158346513,1717155971360126100,1717155971567371181,1717155971734164693,1717155971909329432,1717155972121329457,1717155972316321700,1717155972519983669,1717155972711088569,1717155972909312935,1717155973126061238,1717155973325867841,1717155973529104401,1717155973777687018,1717155973986292611,1717155939600558965,1717155939726387373,1717155939838759705,1717155939959764314,1717155940097402500,1717155940228568325,1717155940373355834,1717155940505859302,1717155940623319079,1717155940747651921,1717155940872348372,1717155940996274194,1717155941118572762,1717155941254799499,1717155941386604340,1717155941515928746,1717155941636897912],"NumGC":1081071,"NumForcedGC":0,"GCCPUFraction":0.03645096078279739,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438589330,"Frees":438582666},{"Size":16,"Mallocs":6789164532,"Frees":6788805307},{"Size":24,"Mallocs":19194153262,"Frees":19194002110},{"Size":32,"Mallocs":10388927383,"Frees":10388840878},{"Size":48,"Mallocs":1995600755,"Frees":1995552754},{"Size":64,"Mallocs":241052732,"Frees":241022095},{"Size":80,"Mallocs":239267862,"Frees":239252623},{"Size":96,"Mallocs":918758075833,"Frees":918756924720},{"Size":112,"Mallocs":384198694,"Frees":384194731},{"Size":128,"Mallocs":9699160630,"Frees":9699145954},{"Size":144,"Mallocs":190748145,"Frees":190743761},{"Size":160,"Mallocs":71674072,"Frees":71672084},{"Size":176,"Mallocs":5504525,"Frees":5503725},{"Size":192,"Mallocs":32451648,"Frees":32451190},{"Size":208,"Mallocs":68396029,"Frees":68392048},{"Size":224,"Mallocs":3202387,"Frees":3201861},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52469955,"Frees":52442402},{"Size":288,"Mallocs":98871874,"Frees":98853361},{"Size":320,"Mallocs":50028832,"Frees":50025368},{"Size":352,"Mallocs":15175384,"Frees":15174846},{"Size":384,"Mallocs":30512674,"Frees":30512487},{"Size":416,"Mallocs":5938889,"Frees":5936411},{"Size":448,"Mallocs":5902400,"Frees":5898093},{"Size":480,"Mallocs":642396,"Frees":642044},{"Size":512,"Mallocs":39699214,"Frees":39698988},{"Size":576,"Mallocs":21162294,"Frees":21159861},{"Size":640,"Mallocs":1235043,"Frees":1234798},{"Size":704,"Mallocs":18256883,"Frees":18256169},{"Size":768,"Mallocs":6995550,"Frees":6995214},{"Size":896,"Mallocs":7336892,"Frees":7336483},{"Size":1024,"Mallocs":52669907,"Frees":52669713},{"Size":1152,"Mallocs":25044615,"Frees":25044542},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16290037,"Frees":16290008},{"Size":1536,"Mallocs":1804301,"Frees":1804297},{"Size":1792,"Mallocs":12415668,"Frees":12413569},{"Size":2048,"Mallocs":711518,"Frees":711422},{"Size":2304,"Mallocs":23014244,"Frees":23013721},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2040608,"Frees":2040591},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8485587,"Frees":8485564},{"Size":4096,"Mallocs":3543848,"Frees":3543762},{"Size":4864,"Mallocs":8047378,"Frees":8047343},{"Size":5376,"Mallocs":1751639,"Frees":1750095},{"Size":6144,"Mallocs":1316861,"Frees":1316851},{"Size":6528,"Mallocs":6079739,"Frees":6079725},{"Size":6784,"Mallocs":1764835,"Frees":1764759},{"Size":6912,"Mallocs":250348,"Frees":250348},{"Size":8192,"Mallocs":226495,"Frees":226393},{"Size":9472,"Mallocs":1844170,"Frees":1844149},{"Size":9728,"Mallocs":107067,"Frees":107067},{"Size":10240,"Mallocs":1190519,"Frees":1190518},{"Size":10880,"Mallocs":33655,"Frees":33655},{"Size":12288,"Mallocs":273581,"Frees":273577},{"Size":13568,"Mallocs":7844263,"Frees":7844114},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1392444,"Frees":1392414},{"Size":18432,"Mallocs":1194451,"Frees":1194446}]} 2024-05-31T11:46:14.405Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:46:14.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:46:14.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:46:14.441Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:46:14.521Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:46:14.524Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:14.542Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:46:14.542Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:46:14.542Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:46:14.545Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:14.572Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:46:14.600Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:15.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:15.541Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:16.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:16.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:16.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:16.578Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:17.297Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:17.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:18.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:19.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:19.605Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:20.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:20.440Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:21.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:21.442Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:22.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:22.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:23.239Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:23.389Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155982492946867, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:46:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:23.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:46:23.389Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:46:23.390Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=387.544µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:46:23.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:24.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:24.410Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:46:24.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:46:24.410Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:46:24.640Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:25.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:25.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:26.286Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:26.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:26.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:26.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:27.348Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:27.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:28.151Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:28.528Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:29.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:29.402Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:46:29.430Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:46:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:46:29.516Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:46:29.516Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:46:29.516Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:46:29.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:29.554Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:29.574Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:46:30.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:30.455Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:31.147Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717155990243086720, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:46:31.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:31.147Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:31.147Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:46:31.148Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:46:31.148Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=465.504µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:46:31.318Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:31.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:32.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:32.479Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:33.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:33.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:34.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:34.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:46:34.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:46:34.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:46:34.587Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:35.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:35.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:36.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:36.329Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:36.330Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:36.479Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:37.308Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:37.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:38.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:38.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:39.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:39.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:40.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:40.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:41.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:41.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:42.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:42.531Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:43.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:43.528Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:44.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:44.402Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:46:44.404Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:46:44.404Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:46:44.404Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:46:44.501Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:44.511Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:46:44.511Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:46:44.511Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:46:44.528Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:44.530Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:46:44.600Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:46:44.617Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:45.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:45.534Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:46.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:46.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:46.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:46.463Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:47.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:47.473Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:48.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:48.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:49.251Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:49.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:50.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:50.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:51.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:51.539Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:52.160Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:52.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:53.248Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:53.419Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156012562846620, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:46:53.419Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:53.419Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:46:53.419Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:46:53.419Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:46:53.419Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=392.528µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:46:53.534Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:54.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:54.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:46:54.402Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:46:54.402Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:46:54.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:55.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:55.568Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:56.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:56.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:56.373Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:46:56.639Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:57.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:57.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:58.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:58.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:59.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:46:59.427Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:46:59.501Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:46:59.501Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:46:59.520Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:46:59.520Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:46:59.520Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:46:59.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:46:59.555Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:46:59.590Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:47:00.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:00.469Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:01.136Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156020297367966, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:47:01.136Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:01.136Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:01.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:47:01.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:47:01.137Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=525.294µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:47:01.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:01.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:02.191Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:02.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:03.184Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:03.686Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:04.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:04.410Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:47:04.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:47:04.410Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:47:04.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:05.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:05.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:06.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:06.375Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:06.388Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:06.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:07.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:07.458Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:08.301Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:08.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:09.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:09.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:10.210Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:10.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:11.272Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:11.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:12.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:12.475Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:13.311Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:13.528Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:14.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:14.413Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:47:14.427Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:47:14.427Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:47:14.427Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:47:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:47:14.500Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:47:14.577Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:47:14.670Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:15.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:15.458Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:16.286Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:16.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:16.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:16.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:17.305Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:17.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:18.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:18.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:19.340Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:19.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:20.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:20.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:21.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:21.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:22.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:22.523Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:23.285Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156042523852856, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:47:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:47:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:47:23.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=408.149µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:47:23.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:24.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:24.431Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:47:24.432Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:47:24.432Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:47:24.588Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:25.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:25.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:26.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:26.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:26.336Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:26.590Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:27.210Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:27.547Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:28.182Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:28.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:29.320Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:29.416Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:47:29.475Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:47:29.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:47:29.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:29.541Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:47:30.170Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:30.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:31.137Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156050179265229, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:47:31.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:31.139Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:31.139Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:47:31.140Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:47:31.141Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=635.173µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:47:31.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:31.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:32.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:32.456Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:33.290Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:33.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:34.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:34.411Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:47:34.411Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:47:34.411Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:47:34.581Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:35.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:35.455Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:36.309Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:36.366Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:36.366Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:37.236Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:37.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:37.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:38.228Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:38.530Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:39.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:39.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:40.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:40.634Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:41.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:41.488Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:42.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:42.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:43.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:43.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:44.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:44.410Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:47:44.410Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:47:44.410Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:47:44.410Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:47:44.463Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:47:44.538Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:47:44.553Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:47:44.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:45.263Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:45.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:46.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:46.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:46.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:46.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:47.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:47.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:48.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:48.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:49.229Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:49.487Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:50.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:50.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:51.335Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:51.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:52.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:52.520Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:53.161Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:53.398Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156072522124721, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:47:53.398Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:53.398Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:47:53.398Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:47:53.398Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:47:53.399Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=893.277µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:47:53.522Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:54.315Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:47:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:47:54.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:47:54.493Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:55.283Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:55.513Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:56.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:56.343Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:56.356Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:47:56.531Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:57.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:57.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:58.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:58.595Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:59.275Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:47:59.398Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:47:59.444Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:47:59.466Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:47:59.489Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:47:59.503Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:48:00.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:00.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156080233134542, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:48:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:48:01.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:48:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=496.095µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:48:01.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:01.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:02.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:02.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:03.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:03.576Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:04.292Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:04.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:48:04.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:48:04.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:48:04.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:05.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:05.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:06.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:06.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:06.360Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:06.553Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:07.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:08.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:08.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:09.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:09.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:10.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:10.598Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:11.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:11.483Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:12.302Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:12.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:13.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:13.525Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:14.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:14.410Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":171851424,"TotalAlloc":91409473406936,"Sys":327718216,"Lookups":0,"Mallocs":973719755356,"Frees":973718002569,"HeapAlloc":171851424,"HeapSys":308510720,"HeapIdle":114089984,"HeapInuse":194420736,"HeapReleased":65839104,"HeapObjects":1752787,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3255040,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3194170,"GCSys":7331136,"OtherSys":1609054,"NextGC":220604752,"LastGC":1717156094048741498,"PauseTotalNs":3797437483447,"PauseNs":[2912718,12149770,16193914,18857517,12507089,8037949,8396962,4665362,8178280,30265197,13454252,135120,32331191,14591210,134740,25204973,16172374,17365959,10149270,33933374,152221,97409,22420528,7456512,7965855,35366196,149839,15543259,11558275,13759991,7189006,8161537,4112970,22584516,11324944,3654201,8389526,121543,8135666,7046099,11815377,123809,15734617,13551221,146256,10624087,14253174,24029269,146073,14795527,9363739,127973,141307,17340382,20201241,141830,7935270,1898538,4180156,12212364,30463499,12371387,26595176,13386811,18347913,8440891,3904578,10300291,4134785,12021556,682249,18587169,6791198,26266218,21811906,15471253,16337050,5266722,13177519,11827969,14385863,23946111,29414502,18968009,8338652,12181242,16236410,15080552,9632637,21509184,317070,7864487,18580215,7632643,5887683,9685745,14090199,17659462,143570,16119250,137820,17796991,21210043,16557862,20194411,13111888,14243252,158946,20298923,12029939,31491149,21448267,166135,94351,21931426,7399303,10450282,3992893,4137334,19156315,12103501,143725,168626,12841660,8263859,37346828,19264748,12280319,24447609,14681008,10922882,21788798,26352095,192588,12231745,18211317,9856873,14407825,11689647,130282,22405392,8167421,150192,90134,20896731,17276592,8149313,14737782,25833958,12236233,14100108,11268286,16059101,21923023,12601787,19909209,12531125,20465883,168669,122189,129831,9237771,4837836,7641408,15969757,18643319,149558,29440261,7498776,144658,26002148,133984,19238010,1300880,9021440,8697972,19937094,96513,12109920,12557060,6510373,15971592,16806669,17479422,13200170,24843342,22459258,20250014,17447827,16030287,157030,19591476,11366245,288764,129988,122353,163011,108089,117491,114662,6935636,112435,133281,128536,129910,130569,121271,131475,127148,132119,300390,150833,127991,120640,5721588,7372779,293076,138439,124985,121857,7736014,130970,147882,106818,411158,146680,302879,117219,4962124,294743,286909,298665,141870,132795,143337,143482,125709,11331563,171342,4141307,20067095,23103084,8083636,17555204,12202926,4084158,10951089,339192,10706561,8440235,7441837,9940297,216119,8413975,1210867,9431841],"PauseEnd":[1717155983947353045,1717155984554374011,1717155985053298149,1717155985681055829,1717155986182512508,1717155986740989701,1717155987291166549,1717155987870832390,1717155988409829399,1717155989065064550,1717155989643147680,1717155990099229985,1717155990677488814,1717155991195973695,1717155991707606181,1717155992202348169,1717155992793495328,1717155993343839401,1717155993974157850,1717155994586921610,1717155995072482864,1717155995692889705,1717155996225632563,1717155996774636883,1717155997331614428,1717155997956680062,1717155998519158843,1717155999154522062,1717155999687696593,1717156000204014346,1717156000750967119,1717156001333925930,1717156001854664333,1717156002379439148,1717156002848187991,1717156003369708688,1717156003962212090,1717156004528094488,1717156005035337970,1717156005632540969,1717156006156832333,1717156006698338316,1717156007246422712,1717156007839279000,1717156008398996265,1717156009057773934,1717156009670968090,1717156010239116973,1717156010772236264,1717156011385807997,1717156011907005380,1717156012443599956,1717156012966589090,1717156013603666999,1717156014321746313,1717156014865195707,1717156015465246689,1717156015975054068,1717156016590527318,1717156017085864994,1717156017689254495,1717156018251406539,1717156018929016797,1717156019668261815,1717156020241974604,1717156020771843875,1717156021401277518,1717156021902595990,1717156022483921267,1717156022988897479,1717156023636906355,1717156024377627483,1717156024907778491,1717156025452627412,1717156025950460819,1717156026548227166,1717156027062163794,1717156027662978396,1717156028178243472,1717156028808201969,1717156029466371089,1717156030003792737,1717156030601185685,1717156031114651673,1717156031684116947,1717156032165046703,1717156032706531725,1717156033199354282,1717156033913184824,1717156034635697696,1717156035175541788,1717156035698384856,1717156036250708957,1717156036788446310,1717156037395437776,1717156037966466834,1717156038554035996,1717156039286436476,1717156039862070025,1717156040440372237,1717156040906816335,1717156041531722691,1717156042030226716,1717156042633756134,1717156043200502031,1717156043801922774,1717156044519908320,1717156045037529320,1717156045636043985,1717156046141081761,1717156046815258857,1717156047453660816,1717156047986989974,1717156048593321047,1717156049384083838,1717156049924275814,1717156050516100590,1717156051025654078,1717156051678953282,1717156052208305039,1717156052733873721,1717156053291002122,1717156053933832217,1717156054616050928,1717156055105120123,1717156055666603864,1717156056192009109,1717156056817742066,1717156057480105927,1717156058034732843,1717156058634122760,1717156059337815256,1717156059928797068,1717156060487087968,1717156061053468771,1717156061684344323,1717156062247150503,1717156062830324603,1717156063347438433,1717156063947018081,1717156064632040810,1717156065184943079,1717156065713989339,1717156066284749891,1717156066825487304,1717156067392852228,1717156067926633248,1717156068448437808,1717156069071431287,1717156069650741412,1717156070193003567,1717156070743174846,1717156071304368715,1717156071858258837,1717156072415644689,1717156072985080476,1717156073587541811,1717156074227782449,1717156074734163672,1717156075326345045,1717156075828324787,1717156076385285900,1717156076856789723,1717156077424136024,1717156077962611507,1717156078550046789,1717156079158461163,1717156079728018799,1717156080290268430,1717156080820697196,1717156081409200552,1717156081883726163,1717156082459461688,1717156082960573140,1717156083537895645,1717156084236418161,1717156084802956211,1717156085417615846,1717156085936010509,1717156086521420932,1717156087007442569,1717156087562317423,1717156088084857260,1717156088647404320,1717156089353563857,1717156089904650023,1717156090535481178,1717156091057856646,1717156091688128087,1717156092243417223,1717156092779036527,1717156093373621223,1717156094048741498,1717155966164349037,1717155966302155701,1717155966441379465,1717155966571493990,1717155966709719659,1717155966847655675,1717155966972857271,1717155967121408502,1717155967255884493,1717155967409428498,1717155967541502879,1717155967670711791,1717155967799436849,1717155967935561326,1717155968069424649,1717155968214354242,1717155968369737636,1717155968522676035,1717155968689284182,1717155968841417381,1717155968980781031,1717155969180070585,1717155969399222048,1717155969596146809,1717155969780876091,1717155969969508795,1717155970167911464,1717155970383124785,1717155970580572259,1717155970764645197,1717155970936905209,1717155971158346513,1717155971360126100,1717155971567371181,1717155971734164693,1717155971909329432,1717155972121329457,1717155972316321700,1717155972519983669,1717155972711088569,1717155972909312935,1717155973126061238,1717155973325867841,1717155973529104401,1717155973777687018,1717155973986292611,1717155974492865731,1717155975007177008,1717155975606006534,1717155976068371414,1717155976655762375,1717155977242594546,1717155977861327437,1717155978375654581,1717155979014283729,1717155979604731518,1717155980111142449,1717155980666091962,1717155981130755538,1717155981678884743,1717155982228124435,1717155982798407341,1717155983340182758],"NumGC":1081281,"NumForcedGC":0,"GCCPUFraction":0.036462913463625915,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438695653,"Frees":438689030},{"Size":16,"Mallocs":6790893868,"Frees":6790532265},{"Size":24,"Mallocs":19199231328,"Frees":19199069721},{"Size":32,"Mallocs":10391690316,"Frees":10391596780},{"Size":48,"Mallocs":1996150604,"Frees":1996101245},{"Size":64,"Mallocs":241095619,"Frees":241065003},{"Size":80,"Mallocs":239308872,"Frees":239293632},{"Size":96,"Mallocs":918966907203,"Frees":918965977832},{"Size":112,"Mallocs":384289881,"Frees":384285918},{"Size":128,"Mallocs":9701787375,"Frees":9701766135},{"Size":144,"Mallocs":190791753,"Frees":190787366},{"Size":160,"Mallocs":71684537,"Frees":71682549},{"Size":176,"Mallocs":5505070,"Frees":5504268},{"Size":192,"Mallocs":32460444,"Frees":32460001},{"Size":208,"Mallocs":68411782,"Frees":68407791},{"Size":224,"Mallocs":3202710,"Frees":3202184},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52479143,"Frees":52451588},{"Size":288,"Mallocs":98877843,"Frees":98859327},{"Size":320,"Mallocs":50035399,"Frees":50031935},{"Size":352,"Mallocs":15178030,"Frees":15177487},{"Size":384,"Mallocs":30521191,"Frees":30521019},{"Size":416,"Mallocs":5940257,"Frees":5937779},{"Size":448,"Mallocs":5902426,"Frees":5898119},{"Size":480,"Mallocs":642396,"Frees":642044},{"Size":512,"Mallocs":39710013,"Frees":39709766},{"Size":576,"Mallocs":21163642,"Frees":21161209},{"Size":640,"Mallocs":1235297,"Frees":1235052},{"Size":704,"Mallocs":18261306,"Frees":18260592},{"Size":768,"Mallocs":6997403,"Frees":6997077},{"Size":896,"Mallocs":7338884,"Frees":7338480},{"Size":1024,"Mallocs":52683545,"Frees":52683335},{"Size":1152,"Mallocs":25047215,"Frees":25047141},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16294441,"Frees":16294412},{"Size":1536,"Mallocs":1804797,"Frees":1804793},{"Size":1792,"Mallocs":12419057,"Frees":12416973},{"Size":2048,"Mallocs":711687,"Frees":711590},{"Size":2304,"Mallocs":23016269,"Frees":23015745},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2041000,"Frees":2040983},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8488101,"Frees":8488093},{"Size":4096,"Mallocs":3544611,"Frees":3544522},{"Size":4864,"Mallocs":8048129,"Frees":8048092},{"Size":5376,"Mallocs":1751707,"Frees":1750163},{"Size":6144,"Mallocs":1316885,"Frees":1316875},{"Size":6528,"Mallocs":6081587,"Frees":6081583},{"Size":6784,"Mallocs":1765501,"Frees":1765430},{"Size":6912,"Mallocs":250408,"Frees":250408},{"Size":8192,"Mallocs":226543,"Frees":226441},{"Size":9472,"Mallocs":1844799,"Frees":1844774},{"Size":9728,"Mallocs":107113,"Frees":107113},{"Size":10240,"Mallocs":1190759,"Frees":1190758},{"Size":10880,"Mallocs":33667,"Frees":33667},{"Size":12288,"Mallocs":273649,"Frees":273645},{"Size":13568,"Mallocs":7846777,"Frees":7846626},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1392836,"Frees":1392806},{"Size":18432,"Mallocs":1194892,"Frees":1194883}]} 2024-05-31T11:48:14.410Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:48:14.412Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:48:14.412Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:48:14.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:48:14.432Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:48:14.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:14.480Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:48:14.495Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:48:15.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:15.500Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:16.262Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:16.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:16.363Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:16.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:17.244Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:17.539Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:18.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:18.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:19.240Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:19.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:20.345Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:20.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:21.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:21.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:22.165Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:22.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:23.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:23.397Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156102555414299, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:48:23.397Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:23.397Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:23.398Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:48:23.398Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:48:23.398Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=440.983µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:48:23.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:24.264Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:48:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:48:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:48:24.444Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:25.293Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:25.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:26.224Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:26.379Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:26.380Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:26.517Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:27.150Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:27.565Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:28.195Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:28.585Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:29.306Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:29.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:48:29.433Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:48:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:48:29.505Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:48:29.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:30.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:30.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156110223706442, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:48:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:31.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:48:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:48:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=474.183µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:48:31.200Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:31.552Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:32.194Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:32.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:33.176Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:33.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:34.304Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:48:34.396Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:48:34.396Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:48:34.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:35.295Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:35.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:36.163Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:36.371Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:36.372Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:36.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:37.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:37.614Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:38.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:38.463Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:39.246Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:39.541Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:40.190Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:40.546Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:41.166Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:41.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:42.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:42.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:43.280Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:43.469Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:44.218Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:44.466Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:48:44.466Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:48:44.466Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:48:44.466Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:48:44.497Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:48:44.516Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:48:44.552Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:48:44.617Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:45.231Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:45.461Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:46.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:46.344Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:46.353Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:46.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:47.225Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:47.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:48.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:48.563Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:49.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:49.464Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:50.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:50.486Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:51.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:51.562Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:52.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:52.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:53.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:53.397Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156132570492475, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:48:53.398Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:53.398Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:48:53.398Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:48:53.398Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:48:53.398Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=373.416µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:48:53.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:54.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:48:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:48:54.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:48:54.516Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:55.214Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:55.514Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:56.202Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:56.367Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:56.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:48:56.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:57.247Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:57.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:58.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:58.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:48:59.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:48:59.408Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:48:59.462Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:48:59.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:48:59.496Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:48:59.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:00.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:00.503Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:01.149Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156140251509139, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:49:01.149Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:01.149Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:01.149Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:49:01.149Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:49:01.150Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=502.246µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:49:01.410Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:01.709Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:02.330Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:02.616Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:03.146Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:03.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:04.327Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:04.402Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:49:04.403Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:49:04.403Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:49:04.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:05.262Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:05.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:06.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:06.376Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:06.376Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:06.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:07.162Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:07.594Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:08.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:08.548Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:09.287Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:09.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:10.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:10.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:11.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:11.571Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:12.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:12.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:13.234Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:13.558Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:14.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:14.402Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:49:14.405Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:49:14.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:49:14.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:49:14.440Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:49:14.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:49:14.507Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:49:14.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:15.220Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:15.520Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:16.159Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:16.393Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:16.393Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:16.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:17.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:17.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:18.308Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:18.484Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:19.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:19.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:20.204Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:20.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:21.152Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:21.574Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:22.168Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:22.570Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:23.235Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:23.385Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156162583443130, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:49:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:49:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:49:23.386Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=357.967µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:49:23.519Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:24.224Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:49:24.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:49:24.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:49:24.536Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:25.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:25.545Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:26.181Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:26.358Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:26.359Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:26.655Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:27.334Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:27.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:28.172Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:28.480Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:29.243Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:29.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:49:29.476Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:49:29.506Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:49:29.520Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:49:29.595Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:30.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:30.453Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:31.142Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156170243259187, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:49:31.142Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:31.142Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:31.142Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:49:31.142Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:49:31.143Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=523.799µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:49:31.326Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:31.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:32.342Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:32.494Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:33.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:33.537Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:34.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:34.433Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:49:34.433Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:49:34.433Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:49:34.596Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:35.277Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:35.463Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:36.230Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:36.367Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:36.367Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:36.464Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:37.327Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:37.550Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:38.208Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:38.623Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:39.324Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:39.511Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:40.158Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:40.543Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:41.174Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:41.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:42.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:42.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:43.232Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:43.531Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:44.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:44.403Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:49:44.400Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:49:44.406Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:49:44.406Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:49:44.477Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:49:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:49:44.510Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:49:44.569Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:45.180Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:45.560Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:46.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:46.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:46.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:46.655Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:47.316Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:47.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:48.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:48.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:49.223Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:49.503Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:50.265Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:50.468Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:51.312Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:51.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:52.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:52.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:53.196Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:53.390Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156192526695971, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:49:53.390Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:53.390Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:49:53.390Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:49:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:49:53.391Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=621.688µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:49:53.574Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:54.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:49:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:49:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:49:54.477Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:55.299Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:55.463Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:56.367Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:56.368Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:49:56.368Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:56.529Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:57.309Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:57.542Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:58.179Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:58.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:59.227Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:49:59.405Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:49:59.443Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:49:59.485Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:49:59.503Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:49:59.514Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:50:00.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:00.443Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:01.156Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156200289069079, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:50:01.157Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:01.157Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:01.158Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:50:01.159Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:50:01.176Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=16.762239ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:50:01.345Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:01.492Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:02.161Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:02.582Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:03.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:03.445Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:04.255Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:04.422Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:50:04.423Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:50:04.423Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:50:04.544Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:05.198Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:05.567Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:06.219Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:06.333Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:06.334Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:06.535Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:07.259Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:07.462Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:08.278Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:08.509Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:10.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:10.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:11.087Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:11.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:12.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:12.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:13.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:13.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:14.092Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:14.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:14.397Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":160907568,"TotalAlloc":91432883502448,"Sys":327718216,"Lookups":0,"Mallocs":973969901284,"Frees":973968274853,"HeapAlloc":160907568,"HeapSys":308510720,"HeapIdle":124583936,"HeapInuse":183926784,"HeapReleased":94527488,"HeapObjects":1626431,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3064640,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3194170,"GCSys":7331136,"OtherSys":1609054,"NextGC":194311368,"LastGC":1717156214312784294,"PauseTotalNs":3799721472403,"PauseNs":[5405700,17176509,22385287,19133194,20235808,4018358,14943817,2524299,21439710,21420934,20354522,13080967,20164071,12115779,22874221,3702962,14171034,16111800,131469,418613,10120754,12084499,1932706,15582350,16333793,20485357,141651,19043464,16327435,157465,172760,3508079,16677557,140859,28002841,10488617,13697241,21339789,27407197,14299551,7915911,19582629,10199236,25364171,4026262,16083611,166920,11551949,13987919,8506478,12837057,3117035,4046191,18658184,11502744,5044819,8053784,172890,10799621,12173069,12176857,15616654,16212533,802830,4169005,8328054,147728,12171370,13261593,23836310,15470136,18340844,8307118,3618321,151285,11552875,10171725,2375731,8107534,12698048,14930502,9243085,3734855,12181774,202535,27805618,22502815,12302279,1389446,20828046,4033201,153952,25687180,130940,1480944,161202,23275791,8111965,13640625,19271843,21098279,26031765,10776401,2840522,8573181,12153094,15775958,8133664,17502806,12153626,8194282,2309909,25229960,21650634,129287,12968119,147984,3144753,8160472,4380110,8385567,14976159,10612336,8644530,9551141,11976009,152331,16022478,15070819,159543,22980041,11623289,23936109,104205,300602,96481,11585814,7267206,144404,138185,143267,295484,130740,133636,137863,137953,106555,129304,127042,153422,118404,128896,317459,140956,132727,148597,115983,123165,110409,111333,287258,138180,126551,179421,110287,133952,127125,128960,147430,136828,138678,114462,119873,129549,302360,125905,140959,133979,136763,12557060,6510373,15971592,16806669,17479422,13200170,24843342,22459258,20250014,17447827,16030287,157030,19591476,11366245,22253090,12222326,7892399,12915842,15959763,25390433,10557089,11692855,9514914,16803231,8420726,141229,13151082,3200861,16966681,214975,8098006,18994921,120663,16319930,152074,12711433,12229929,16818978,3812693,8028792,23586018,11520165,13067096,21580731,7211882,3986315,17629269,9765669,7827577,20987604,19011164,4800348,11685106,22491722,12167774,10496963,11527018,371025,2524682,9027827,8060656,4363829,126508,12171851,8582223,22952994,18807856,12237284,9207554,15955593,7458435,20063422,16379417,11475056,12147866,22184158,12950966],"PauseEnd":[1717156130260343085,1717156130860267761,1717156131442871036,1717156131949485960,1717156132500425275,1717156132974614690,1717156133610785966,1717156134271775187,1717156134806453032,1717156135385884408,1717156135940189709,1717156136591111443,1717156137097489951,1717156137690945957,1717156138263202829,1717156138818179687,1717156139516820622,1717156140014421198,1717156140622994111,1717156141214400852,1717156142131845897,1717156142945837322,1717156143515896978,1717156144198218764,1717156144760615223,1717156145311631683,1717156145831333454,1717156146394957512,1717156146928644878,1717156147520470688,1717156148045842847,1717156148590980318,1717156149283941600,1717156149795957821,1717156150370288755,1717156150855398760,1717156151432389711,1717156151955750846,1717156152533612983,1717156153036577160,1717156153628697430,1717156154320705888,1717156154854990670,1717156155401600611,1717156155923771571,1717156156507784578,1717156157019011030,1717156157663190414,1717156158182107565,1717156158755555233,1717156159404774682,1717156159882667754,1717156160394199812,1717156160898826410,1717156161457511097,1717156161947994314,1717156162538288160,1717156163050558364,1717156163672586403,1717156164364068022,1717156164891252345,1717156165442590218,1717156165973650003,1717156166625053717,1717156167249437431,1717156167820695207,1717156168334978399,1717156168902896109,1717156169599237512,1717156170114515751,1717156170680978935,1717156171223350130,1717156171761646865,1717156172358639871,1717156172875890942,1717156173447233967,1717156173982754209,1717156174641383425,1717156175142751290,1717156175637152445,1717156176128316426,1717156176740578319,1717156177419283591,1717156177981807285,1717156178548890262,1717156179280414602,1717156179859115487,1717156180410817755,1717156180899314413,1717156181492528377,1717156181962001272,1717156182550050020,1717156183093080070,1717156183656227070,1717156184405377356,1717156184939960360,1717156185498809723,1717156186038218250,1717156186677315497,1717156187207363190,1717156187838296775,1717156188396582673,1717156188971096118,1717156189630655481,1717156190159833595,1717156190727310115,1717156191306482428,1717156191841188198,1717156192387954436,1717156192923231957,1717156193498492019,1717156194118207798,1717156194705732320,1717156195199753969,1717156195706329842,1717156196264713217,1717156196800866010,1717156197308614873,1717156197917264053,1717156198456978560,1717156199045008236,1717156199683409436,1717156200221853268,1717156200739091202,1717156201272468008,1717156201850300626,1717156202472106640,1717156203069184180,1717156203651059266,1717156204370994709,1717156204912144377,1717156205490616387,1717156206021316606,1717156206607414798,1717156207096823119,1717156207738216240,1717156208347010926,1717156208952859761,1717156209155281361,1717156209281542331,1717156209401218734,1717156209531560841,1717156209661031885,1717156209790629147,1717156209920383221,1717156210055115011,1717156210175557766,1717156210296046670,1717156210421809620,1717156210546118546,1717156210671052293,1717156210803497428,1717156210933135314,1717156211076442200,1717156211208209083,1717156211358944514,1717156211499035420,1717156211628049127,1717156211765498489,1717156211885477775,1717156212007192958,1717156212122047043,1717156212260152258,1717156212395392042,1717156212525232739,1717156212653089691,1717156212774294421,1717156212904851618,1717156213034258039,1717156213153863543,1717156213278961430,1717156213402806408,1717156213532040638,1717156213652053461,1717156213793915448,1717156213931852255,1717156214069071284,1717156214191746236,1717156214312784294,1717156086521420932,1717156087007442569,1717156087562317423,1717156088084857260,1717156088647404320,1717156089353563857,1717156089904650023,1717156090535481178,1717156091057856646,1717156091688128087,1717156092243417223,1717156092779036527,1717156093373621223,1717156094048741498,1717156094675617784,1717156095230598731,1717156095765354766,1717156096295056191,1717156096819062695,1717156097429823182,1717156097930152303,1717156098480264875,1717156099120077350,1717156099711181600,1717156100258203295,1717156100809272903,1717156101380025871,1717156101877988715,1717156102452792362,1717156102943406159,1717156103545973709,1717156104208698227,1717156104724489562,1717156105292436863,1717156105826233957,1717156106403517594,1717156106884949187,1717156107458776755,1717156108005070851,1717156108567846415,1717156109287321059,1717156109842926855,1717156110361293410,1717156110895514553,1717156111464660460,1717156111933179135,1717156112488602517,1717156112967855781,1717156113530262467,1717156114222607156,1717156114773330749,1717156115314157289,1717156115882310739,1717156116512548816,1717156117005584714,1717156117644000764,1717156118162294116,1717156118705221196,1717156119377551934,1717156119873537856,1717156120479411624,1717156120970983335,1717156121580657199,1717156122083449208,1717156122643766184,1717156123186405954,1717156123842854958,1717156124550695671,1717156125063060278,1717156125674979110,1717156126237727882,1717156126792483558,1717156127339251741,1717156127902322797,1717156128465243680,1717156129160370717,1717156129731214838],"NumGC":1081523,"NumForcedGC":0,"GCCPUFraction":0.03647419555050409,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438802886,"Frees":438796262},{"Size":16,"Mallocs":6792633034,"Frees":6792274382},{"Size":24,"Mallocs":19204314908,"Frees":19204165310},{"Size":32,"Mallocs":10394450498,"Frees":10394364010},{"Size":48,"Mallocs":1996700299,"Frees":1996652324},{"Size":64,"Mallocs":241138702,"Frees":241108085},{"Size":80,"Mallocs":239349913,"Frees":239334664},{"Size":96,"Mallocs":919202373989,"Frees":919201540932},{"Size":112,"Mallocs":384381066,"Frees":384377099},{"Size":128,"Mallocs":9704411382,"Frees":9704396717},{"Size":144,"Mallocs":190835379,"Frees":190830993},{"Size":160,"Mallocs":71695005,"Frees":71693017},{"Size":176,"Mallocs":5505619,"Frees":5504819},{"Size":192,"Mallocs":32469321,"Frees":32468868},{"Size":208,"Mallocs":68427731,"Frees":68423751},{"Size":224,"Mallocs":3203035,"Frees":3202509},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52488363,"Frees":52460817},{"Size":288,"Mallocs":98883863,"Frees":98865362},{"Size":320,"Mallocs":50041973,"Frees":50038512},{"Size":352,"Mallocs":15180695,"Frees":15180162},{"Size":384,"Mallocs":30529787,"Frees":30529605},{"Size":416,"Mallocs":5941616,"Frees":5939138},{"Size":448,"Mallocs":5902452,"Frees":5898145},{"Size":480,"Mallocs":642396,"Frees":642044},{"Size":512,"Mallocs":39720814,"Frees":39720601},{"Size":576,"Mallocs":21164996,"Frees":21162565},{"Size":640,"Mallocs":1235553,"Frees":1235308},{"Size":704,"Mallocs":18265713,"Frees":18264999},{"Size":768,"Mallocs":6999298,"Frees":6998964},{"Size":896,"Mallocs":7340888,"Frees":7340482},{"Size":1024,"Mallocs":52697217,"Frees":52697040},{"Size":1152,"Mallocs":25049827,"Frees":25049756},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16298845,"Frees":16298816},{"Size":1536,"Mallocs":1805291,"Frees":1805287},{"Size":1792,"Mallocs":12422502,"Frees":12420408},{"Size":2048,"Mallocs":711857,"Frees":711761},{"Size":2304,"Mallocs":23018306,"Frees":23017785},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2041395,"Frees":2041378},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8490673,"Frees":8490655},{"Size":4096,"Mallocs":3545379,"Frees":3545295},{"Size":4864,"Mallocs":8048886,"Frees":8048851},{"Size":5376,"Mallocs":1751775,"Frees":1750231},{"Size":6144,"Mallocs":1316909,"Frees":1316899},{"Size":6528,"Mallocs":6083482,"Frees":6083470},{"Size":6784,"Mallocs":1766178,"Frees":1766105},{"Size":6912,"Mallocs":250468,"Frees":250468},{"Size":8192,"Mallocs":226592,"Frees":226490},{"Size":9472,"Mallocs":1845431,"Frees":1845410},{"Size":9728,"Mallocs":107161,"Frees":107161},{"Size":10240,"Mallocs":1191000,"Frees":1190999},{"Size":10880,"Mallocs":33679,"Frees":33679},{"Size":12288,"Mallocs":273717,"Frees":273713},{"Size":13568,"Mallocs":7849349,"Frees":7849204},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1393230,"Frees":1393200},{"Size":18432,"Mallocs":1195333,"Frees":1195328}]} 2024-05-31T11:50:14.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:50:14.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:50:14.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:50:14.397Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:50:14.414Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:50:14.429Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:50:14.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:50:14.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:50:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:50:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:50:14.500Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:50:15.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:15.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:16.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:16.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:16.327Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:16.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:17.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:17.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:18.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:18.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:19.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:19.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:20.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:20.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:21.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:21.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:22.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:22.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:23.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:23.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:23.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156223359301587, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:50:23.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:23.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:23.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:50:23.392Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:50:23.392Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=391.974µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:50:24.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:24.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:24.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:50:24.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:50:24.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:50:25.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:25.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:26.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:26.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:26.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:26.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:27.078Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:27.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:28.091Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:28.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:29.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:29.366Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:50:29.412Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:50:29.426Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:50:29.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:50:30.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:30.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:31.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:31.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156231072833552, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:50:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:31.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:50:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:50:31.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=521.622µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:50:31.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:32.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:32.382Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:33.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:33.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:34.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:34.377Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:34.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:50:34.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:50:34.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:50:35.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:35.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:36.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:36.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:36.338Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:36.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:37.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:37.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:38.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:38.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:39.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:39.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:40.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:40.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:41.076Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:41.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:42.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:42.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:43.083Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:43.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:44.084Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:44.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:44.399Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:50:44.400Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:50:44.400Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:50:44.400Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:50:44.416Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:50:44.434Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:50:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:50:45.088Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:45.358Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:46.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:46.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:46.328Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:46.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:47.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:47.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:48.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:48.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:49.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:49.365Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:50.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:50.360Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:51.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:51.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:52.100Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:52.359Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:53.072Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:53.356Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:53.391Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156253357436278, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:50:53.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:53.391Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:50:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:50:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:50:53.392Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=390.03µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:50:54.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:54.364Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:54.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:50:54.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:50:54.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:50:55.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:55.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:56.081Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:56.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:56.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:50:56.376Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:57.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:57.374Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:58.082Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:58.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:59.075Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:50:59.370Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:50:59.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:50:59.411Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:50:59.425Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:50:59.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:00.070Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:00.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:01.069Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:01.133Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156261070196994, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:51:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:01.133Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:51:01.133Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:51:01.134Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=1.052198ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:51:01.367Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:02.073Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:02.362Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:03.077Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:03.361Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:04.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:04.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:04.401Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:51:04.401Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:51:04.401Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:51:05.071Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:05.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:06.086Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:06.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:06.337Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:06.380Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:07.080Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:07.357Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:08.079Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:08.381Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:09.102Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:09.396Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:10.118Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:10.378Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:11.096Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:11.392Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:12.137Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:13.124Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:13.382Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:14.284Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:14.402Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:51:14.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:51:14.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:51:14.429Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:51:14.430Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:51:14.493Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:51:14.496Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:51:14.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:14.499Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:51:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:51:14.499Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:51:14.510Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:15.217Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:15.504Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:16.173Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:16.352Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:16.372Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:16.555Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:17.216Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:17.549Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:18.237Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:18.469Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:19.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:19.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:20.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:20.600Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:21.164Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:21.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:22.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:22.502Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:23.282Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:23.386Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156282503117586, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:51:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:23.386Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:51:23.386Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:51:23.387Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=381.916µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:51:23.503Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:24.363Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:24.427Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:51:24.427Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:51:24.427Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:51:24.580Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:25.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:25.571Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:26.341Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:26.342Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:26.629Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:26.970Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:27.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:27.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:28.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:28.620Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:29.328Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:51:29.435Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:51:29.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:29.511Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:51:29.512Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:51:29.513Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:51:29.513Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:51:29.526Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:29.530Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:30.238Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:30.478Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:31.134Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156290251263371, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:51:31.135Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:31.135Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:31.136Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:51:31.137Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:51:31.138Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=543.995µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:51:31.201Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:31.501Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:32.187Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:32.554Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:33.189Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:33.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:33.819Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Start checkpointing for replication 2024-05-31T11:51:34.288Z INFO GOXDCR.CheckpointSvc: GetMappingsDoc f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 retrieved an existing mappingsDoc 2024-05-31T11:51:34.290Z INFO GOXDCR.CheckpointMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Checkpointing for vb_list=[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511], time_to_wait=10m0s, interval_btwn_vb=1.171 sec 2024-05-31T11:51:34.372Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:34.419Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:51:34.419Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:51:34.419Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:51:34.577Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:35.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:35.561Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:36.199Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:36.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:36.340Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:36.624Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:37.343Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:37.566Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:38.171Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:38.590Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:39.316Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:39.498Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:40.339Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:40.496Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:41.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:41.506Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:42.183Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:42.551Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:43.215Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:43.532Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:44.268Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:44.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:51:44.397Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:51:44.397Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:51:44.397Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:51:44.465Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:51:44.492Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.9.74:8091 ] [10.0.15.212:8091 ]] 2024-05-31T11:51:44.497Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:51:44.504Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.9.74:8091 and ActiveHttpsHostName with 2024-05-31T11:51:44.512Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:51:44.512Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:51:44.517Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:44.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:45.270Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:45.471Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:46.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:46.356Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:46.372Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:46.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:47.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:47.573Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:48.188Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:48.599Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:49.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:49.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:50.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:50.524Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:51.185Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:51.571Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:52.205Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:52.656Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:53.212Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:53.388Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156312656807928, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:51:53.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:53.389Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:51:53.390Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:51:53.391Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:51:53.391Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=359.734µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:51:53.510Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:54.384Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:54.398Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:51:54.398Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:51:54.398Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:51:54.591Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:55.192Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:55.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:56.203Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:56.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:56.347Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:51:56.490Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:57.237Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:57.452Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:58.316Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:58.491Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:51:59.421Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:51:59.428Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:51:59.514Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:51:59.517Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:59.534Z INFO GOXDCR.RemClusterSvc: Replaced ActiveHostName in ref remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= with 10.0.15.212:8091 and ActiveHttpsHostName with 2024-05-31T11:51:59.534Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in checkAndUpdateAgentReference isSame: false, nodeListUpdated: false, addressPrefUpdate: false, capabilityChanged: false, isEssentiallySame: true 2024-05-31T11:51:59.534Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= in updateReferenceFromInternal updateMetaKv: false, ref revision is nil: false 2024-05-31T11:51:59.549Z INFO GOXDCR.RemClusterSvc: Refresher has successfully committed staged changes: Original: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] Actual staged changes now in memory: id:remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA=; uuid:f8411c7deed0a1c9dd99daeaa4e5fbde; name:c2; hostName:10.0.15.212:8091; userName:Administrator; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 52 125 195 223 14 108 0 0 0 2 104 2 109 0 0 0 32 51 102 54 51 97 57 51 101 55 50 50 48 98 57 52 50 51 49 52 48 57 102 98 56 50 50 53 52 49 102 57 53 104 2 97 2 110 5 0 173 218 196 223 14 104 2 109 0 0 0 32 51 102 55 51 56 98 97 51 57 98 52 55 52 52 57 53 54 49 50 52 52 51 48 101 101 52 100 102 99 50 57 57 104 2 97 1 110 5 0 173 218 196 223 14 106] [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:51:59.566Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:51:59.584Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:00.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:00.586Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:01.136Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156320169846368, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:52:01.137Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:52:01.138Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:52:01.138Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:52:01.139Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:52:01.156Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=16.845449ms num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:52:01.250Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:01.512Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:02.334Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:02.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:03.371Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:03.505Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:04.222Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:04.412Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:52:04.412Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:52:04.412Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:52:04.592Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:05.226Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:05.557Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:06.257Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:06.353Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:06.353Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:06.507Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:07.291Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:07.453Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:08.288Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:08.474Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:09.242Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:09.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:10.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:10.664Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:11.350Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:11.499Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:12.331Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:12.489Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:13.154Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:13.538Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:14.269Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:14.396Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:52:14.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:52:14.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:52:14.407Z INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":171739352,"TotalAlloc":91481797255448,"Sys":327718216,"Lookups":0,"Mallocs":974485790066,"Frees":974484048038,"HeapAlloc":171739352,"HeapSys":308510720,"HeapIdle":114466816,"HeapInuse":194043904,"HeapReleased":64995328,"HeapObjects":1742028,"StackInuse":1867776,"StackSys":1867776,"MSpanInuse":3256640,"MSpanSys":5189760,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":3194170,"GCSys":7331136,"OtherSys":1609054,"NextGC":215684240,"LastGC":1717156333987570824,"PauseTotalNs":3801070872507,"PauseNs":[151903,130202,131129,136485,126596,131642,360912,137228,133138,303066,119395,139963,145810,130377,142006,130389,115833,129108,133292,199409,137365,155139,168099,136870,142307,130543,133853,137609,151410,144226,132259,134437,122180,133894,129872,140370,151002,112478,150375,125124,141508,133611,157314,136251,134469,133776,142909,135395,182066,137181,160456,141183,316384,215609,141609,139170,130304,135879,138625,300383,147831,137283,140233,138587,134169,141755,3560095,156297,121813,150464,136102,107588,149993,116794,210483,1626893,311908,302533,167694,113999,133781,132194,133469,152414,113442,173403,114110,618092,138870,2813290,147262,159751,142211,128497,147926,143024,140221,138772,130726,284222,173313,1236380,292040,8184889,309331,143302,147278,290997,151882,139809,139505,152325,3273222,111801,137995,1326277,9115416,8491549,8129509,3683758,22920375,16231941,9387623,10670987,7179648,16278722,12522574,20979515,13696261,10609774,184222,19996974,14936430,11557205,4139440,9988792,25039680,24340577,28544681,9259899,9873545,3985573,152433,115864,24759188,1443409,100218,10324044,12130989,338301,6794658,17666743,12118299,18873572,25889346,3501279,21999462,2391236,10753575,123607,4105928,19771377,8120978,18493256,114712,16152196,5555399,16702943,13176575,14890098,36323633,168832,10571569,12097625,7992223,8303641,5108669,8805214,13993818,11511322,3796227,27840744,16546381,3137590,3894284,4084529,22180518,11600891,37538471,16558209,202211,374801,11130912,12109483,7807499,17509649,13477600,1560619,17699919,12227869,4102477,23339615,16142347,145866,188179,22980482,162637,10231364,11223298,16115701,7399628,17640336,302255,14264269,12510159,31153193,226358,12837742,17833845,15467179,168228,15359658,28320784,6003349,136002,130206,106688,130623,137258,123509,239820,134142,127439,126796,207084,124265,127840,166530,134385,140154,150823,136026,154330,126888,290908,135246,134280,130213,128044,294987,145833,112692,153541,135258,131596,153002],"PauseEnd":[1717156256607365728,1717156256736321376,1717156256865998542,1717156256990436216,1717156257119103321,1717156257246332787,1717156257385921539,1717156257523888016,1717156257654276883,1717156257786527105,1717156257916007187,1717156258054605693,1717156258185811946,1717156258312832885,1717156258435168942,1717156258565241199,1717156258693223582,1717156258843009748,1717156258977566982,1717156259104160059,1717156259229582325,1717156259370447955,1717156259498920949,1717156259613146853,1717156259736872786,1717156259859303440,1717156259984667603,1717156260110488674,1717156260238129624,1717156260380895786,1717156260511773677,1717156260626777429,1717156260752439581,1717156260874130780,1717156260997305166,1717156261122503021,1717156261245597034,1717156261382086611,1717156261511864758,1717156261634165326,1717156261764574370,1717156261895541313,1717156262024821498,1717156262141918508,1717156262264589920,1717156262393506082,1717156262515881749,1717156262639395046,1717156262762938847,1717156262886490386,1717156263015768240,1717156263145627122,1717156263273582455,1717156263402949366,1717156263533661605,1717156263649379847,1717156263778996634,1717156263908502472,1717156264030101691,1717156264168279770,1717156264290644334,1717156264428925052,1717156264564246049,1717156264705745064,1717156264853974994,1717156264986701793,1717156265127229293,1717156265262313220,1717156265409392661,1717156265549983057,1717156265689995281,1717156265828005979,1717156265967714852,1717156266116478857,1717156266256643061,1717156266411084557,1717156266570001688,1717156266718484613,1717156266857974868,1717156266992817850,1717156267131793160,1717156267275870671,1717156267422836825,1717156267555736584,1717156267688814189,1717156267827574345,1717156267971822001,1717156268133246421,1717156268281274119,1717156268423106653,1717156268562646967,1717156268695015056,1717156268838998979,1717156268992889886,1717156269209335271,1717156269429905172,1717156269638828044,1717156269840245792,1717156270025983990,1717156270240614719,1717156270459649519,1717156270656454399,1717156270825659069,1717156271038466076,1717156271275377378,1717156271489632896,1717156271673226191,1717156271865429970,1717156272042951374,1717156272251283566,1717156272478810625,1717156272679487175,1717156272867710974,1717156273069657578,1717156273271147158,1717156273476235088,1717156273649017716,1717156273886961118,1717156274280338792,1717156274814952226,1717156275374233197,1717156275880947370,1717156276400120111,1717156276892419691,1717156277477677181,1717156278064849362,1717156278639034232,1717156279381211063,1717156279899799738,1717156280495153875,1717156280969439333,1717156281567918939,1717156282057816350,1717156282652203689,1717156283210964157,1717156283795172466,1717156284463264796,1717156284970394870,1717156285566303214,1717156286040949227,1717156286697295024,1717156287374098281,1717156287909494685,1717156288511136530,1717156289216907497,1717156289754975412,1717156290291261551,1717156290817905142,1717156291404429815,1717156291874812909,1717156292413519173,1717156292959753703,1717156293506361826,1717156294324385170,1717156294922288323,1717156295525853515,1717156296037330464,1717156296655931171,1717156297362344917,1717156297939868179,1717156298507133608,1717156299207520976,1717156299787110979,1717156300317463145,1717156300820659955,1717156301388646299,1717156301871440799,1717156302421371879,1717156302896486806,1717156303439592130,1717156304098879890,1717156304785370182,1717156305335197263,1717156305858142261,1717156306449576293,1717156306961068400,1717156307480200455,1717156308019191728,1717156308581981022,1717156309261548123,1717156309811114333,1717156310374779797,1717156310879919387,1717156311429248789,1717156311918791454,1717156312470805070,1717156313079666860,1717156313728592039,1717156314461688579,1717156315006578835,1717156315607582528,1717156316100808761,1717156316674596423,1717156317141756762,1717156317718668918,1717156318199389664,1717156318725115861,1717156319445288324,1717156319945883554,1717156320519124967,1717156321020115380,1717156321698426677,1717156322207567499,1717156322745339960,1717156323245936296,1717156323826797698,1717156324536241561,1717156325017611485,1717156325610340196,1717156326142940140,1717156326697017839,1717156327177080999,1717156327739188868,1717156328271335451,1717156328771383169,1717156329473532076,1717156329984695006,1717156330594618545,1717156331128437115,1717156331737861486,1717156332274161333,1717156332856827050,1717156333432195264,1717156333987570824,1717156252541080022,1717156252658445116,1717156252771026178,1717156252892485081,1717156253023216195,1717156253147593904,1717156253284281636,1717156253399451477,1717156253520517319,1717156253645066446,1717156253778902460,1717156253907064414,1717156254028107395,1717156254149009393,1717156254270356148,1717156254392402271,1717156254516962723,1717156254635516474,1717156254753736609,1717156254875225474,1717156254998622154,1717156255121972836,1717156255252018691,1717156255392098897,1717156255527175718,1717156255659164999,1717156255789148911,1717156255919920716,1717156256071511664,1717156256205875372,1717156256341817692,1717156256477817988],"NumGC":1082080,"NumForcedGC":0,"GCCPUFraction":0.03648312144158935,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":438921601,"Frees":438914924},{"Size":16,"Mallocs":6794636176,"Frees":6794274496},{"Size":24,"Mallocs":19209533472,"Frees":19209377798},{"Size":32,"Mallocs":10397248893,"Frees":10397159611},{"Size":48,"Mallocs":1997287769,"Frees":1997238234},{"Size":64,"Mallocs":241184713,"Frees":241153046},{"Size":80,"Mallocs":239394527,"Frees":239379284},{"Size":96,"Mallocs":919703004630,"Frees":919702073963},{"Size":112,"Mallocs":384472259,"Frees":384468294},{"Size":128,"Mallocs":9707044365,"Frees":9707027494},{"Size":144,"Mallocs":190887210,"Frees":190882824},{"Size":160,"Mallocs":71705493,"Frees":71703505},{"Size":176,"Mallocs":5506217,"Frees":5505415},{"Size":192,"Mallocs":32478559,"Frees":32478106},{"Size":208,"Mallocs":68444096,"Frees":68440098},{"Size":224,"Mallocs":3203376,"Frees":3202850},{"Size":240,"Mallocs":2206601,"Frees":2206256},{"Size":256,"Mallocs":52497635,"Frees":52470071},{"Size":288,"Mallocs":98904930,"Frees":98885387},{"Size":320,"Mallocs":50048548,"Frees":50045077},{"Size":352,"Mallocs":15183783,"Frees":15183231},{"Size":384,"Mallocs":30538752,"Frees":30538569},{"Size":416,"Mallocs":5942994,"Frees":5940512},{"Size":448,"Mallocs":5902479,"Frees":5898172},{"Size":480,"Mallocs":642396,"Frees":642044},{"Size":512,"Mallocs":39731848,"Frees":39731601},{"Size":576,"Mallocs":21169930,"Frees":21167497},{"Size":640,"Mallocs":1235810,"Frees":1235565},{"Size":704,"Mallocs":18270143,"Frees":18269429},{"Size":768,"Mallocs":7001470,"Frees":7001136},{"Size":896,"Mallocs":7342986,"Frees":7342580},{"Size":1024,"Mallocs":52710846,"Frees":52710627},{"Size":1152,"Mallocs":25055997,"Frees":25055922},{"Size":1280,"Mallocs":2690479,"Frees":2690469},{"Size":1408,"Mallocs":16303251,"Frees":16303222},{"Size":1536,"Mallocs":1805788,"Frees":1805784},{"Size":1792,"Mallocs":12426825,"Frees":12424731},{"Size":2048,"Mallocs":712034,"Frees":711937},{"Size":2304,"Mallocs":23023902,"Frees":23023377},{"Size":2688,"Mallocs":22977,"Frees":22976},{"Size":3072,"Mallocs":2041789,"Frees":2041772},{"Size":3200,"Mallocs":7,"Frees":7},{"Size":3456,"Mallocs":8493608,"Frees":8493590},{"Size":4096,"Mallocs":3546142,"Frees":3546055},{"Size":4864,"Mallocs":8050134,"Frees":8050097},{"Size":5376,"Mallocs":1751843,"Frees":1750299},{"Size":6144,"Mallocs":1316934,"Frees":1316924},{"Size":6528,"Mallocs":6085647,"Frees":6085635},{"Size":6784,"Mallocs":1766948,"Frees":1766875},{"Size":6912,"Mallocs":250528,"Frees":250528},{"Size":8192,"Mallocs":226641,"Frees":226539},{"Size":9472,"Mallocs":1846047,"Frees":1846023},{"Size":9728,"Mallocs":107207,"Frees":107207},{"Size":10240,"Mallocs":1191240,"Frees":1191238},{"Size":10880,"Mallocs":33691,"Frees":33691},{"Size":12288,"Mallocs":273785,"Frees":273781},{"Size":13568,"Mallocs":7852284,"Frees":7852127},{"Size":14336,"Mallocs":2293,"Frees":2269},{"Size":16384,"Mallocs":1393623,"Frees":1393592},{"Size":18432,"Mallocs":1195759,"Frees":1195751}]} 2024-05-31T11:52:14.407Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:52:14.458Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:52:14.509Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:52:14.534Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:14.568Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:52:15.338Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:15.527Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:16.169Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:16.349Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:16.350Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:16.538Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:17.211Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:17.559Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:18.197Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:18.607Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:19.289Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:19.540Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:20.186Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:20.572Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:21.175Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:21.622Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:22.177Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:22.515Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:23.343Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:23.394Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2-418453448 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 368, "num_failedckpts": 0, "time_committing": {"count": 368, "max": 1342, "mean": 26.97554347826087, "min": 0}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156342515930233, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 2271490, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 5416284912, "data_replicated_uncompress": 5416284912, "datapool_failed_gets": 0, "dcp_datach_length": 6511, "dcp_dispatch_time": 22, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 7272916, "docs_cloned": 0, "docs_failed_cr_source": 9206, "docs_failed_cr_target": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 0, "docs_processed": 7272916, "docs_received_from_dcp": 2179165, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 2169956, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 2179163, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 368, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 1, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 103776, "set_docs_written": 2169956, "set_failed_cr_source": 9206, "set_failed_cr_target": 0, "set_filtered": 0, "set_received_from_dcp": 2179165, "set_target_docs_skipped": 0, "size_rep_queue": 4294933138, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 26, "wtavg_docs_latency": 5, "wtavg_get_doc_latency": 1, "wtavg_merge_latency": 0, "wtavg_meta_latency": 0}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0": {"dcp_datach_length": 2952, "dcp_dispatch_time": {"count": 1095244, "max": 1052, "mean": 20.38, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1095245, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1095245, "system_events_received_from_dcp": 0}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1": {"dcp_datach_length": 3559, "dcp_dispatch_time": {"count": 1083919, "max": 1133, "mean": 25.089, "min": 1}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1083920, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1083920, "system_events_received_from_dcp": 0}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2710949283, "data_replicated_uncompress": 2710949283, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 3813, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1091431, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1095244, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1091431, "max": 49, "mean": 1.563, "min": 0}, "set_docs_cas_changed": 47445, "set_docs_written": 1091431, "set_failed_cr_source": 3813, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147477630, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1091431, "max": 72, "mean": 5.103, "min": 0}, "wtavg_get_doc_latency": {"count": 1119821, "max": 16, "mean": 1.195, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 2705335629, "data_replicated_uncompress": 2705335629, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 5393, "docs_failed_cr_target": 0, "docs_opt_repd": 0, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1078525, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 1083919, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1078525, "max": 166, "mean": 1.95, "min": 0}, "set_docs_cas_changed": 56331, "set_docs_written": 1078525, "set_failed_cr_source": 5393, "set_failed_cr_target": 0, "set_target_docs_skipped": 0, "size_rep_queue": 2147455508, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1078525, "max": 95, "mean": 5.412, "min": 0}, "wtavg_get_doc_latency": {"count": 1114955, "max": 20, "mean": 1.226, "min": 0}, "wtavg_meta_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}} 2024-05-31T11:52:23.394Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1095245 items (1095245 compressed), sent 1142689 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.375914181373935ms, size of last ten batches processed [1,2,1,2,1,1,1,2,5,1], len(batches_ready_queue)=0, resend=918, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=47445, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1119821, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:52:23.394Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1083920 items (1083920 compressed), sent 1140249 items (0 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 3.5210002376673866ms, size of last ten batches processed [9,11,5,1,1,5,15,2,29,5], len(batches_ready_queue)=0, resend=1107, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=56330, to resolve=0, to setback=0, numGetMeta=0,numSubdocGet=1114955, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:52:23.394Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_0 received 1095245 items (1095245 compressed), sent 1095244 items. 2024-05-31T11:52:23.395Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_10.0.2.180:11210_1 received 1083920 items (1083920 compressed), sent 1083919 items. 2024-05-31T11:52:23.395Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2_ThroughSeqnoTracker time_spent=744.192µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:52:23.481Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:24.266Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:24.408Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:52:24.408Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:52:24.408Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:52:24.603Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:25.296Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:25.476Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:26.209Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:26.385Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:26.386Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:26.571Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:27.167Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:27.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:28.225Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:28.472Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:29.339Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:29.396Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2 InternalId: SKCHy_u9RcKYJko3spQn8Q== SourceBucketName: test2 SourceBucketUUID: fec97a951e79a724a626abd9c72d1c8f TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:2 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=fec97a951e79a724a626abd9c72d1c8f 2024-05-31T11:52:29.440Z INFO GOXDCR.PipelineMgr: checking pipeline spec=Id: f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2 InternalId: uIMv2Rqu__AnE0f1j5qlLQ== SourceBucketName: test3 SourceBucketUUID: f2b5bc17672593f4309a3c5942af4df4 TargetClusterUUID: f8411c7deed0a1c9dd99daeaa4e5fbde TargetBucketName: test2 TargetBucketUUID: fdd1f1783b4c9e968b8614b7b9603557 Settings: map[CollectionsMgtMulti:ExplicitMapping: false Mirroring: false Migration: false OSO: true active:true backlogThreshold:50 bandwidth_limit:0 checkpoint_interval:600 ckptSvcCacheEnabled:true colMappingRules:map[] collectionsSkipSrcValidation:false compression_type:3 dcpEnablePurgeRollback:false delAllBackfills:false delSpecificBackfillForVb:-1 dismissEvent:-1 doc_batch_size_kb:2048 failure_restart_interval:10 filterSystemScope:true filter_exp_del:0 filter_expression: filter_expression_version:0 filter_skip_restream:false jsFunctionTimeoutMs:20000 log_level:Info manualBackfill: mergeFunctionMapping:map[] mobile:1 optimistic_replication_threshold:256 preReplicateVBMasterCheck:true priority:High replicateCkptIntervalMin:20 replication_type:xmem retryOnErrExceptAuthErrMaxWaitSec:360 retryOnRemoteAuthErr:true retryOnRemoteAuthErrMaxWaitSec:360 source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500 xdcrDevBackfillReplUpdateDelayMs:0 xdcrDevBackfillRollbackTo0VB:-1 xdcrDevBackfillSendDelayMs:0 xdcrDevBucketTopologyLegacyDelay:0 xdcrDevCkptMgrForceGCWaitSec:0 xdcrDevColManifestSvcDelaySec:0 xdcrDevMainRollbackTo0VB:-1 xdcrDevMainSendDelayMs:0 xdcrDevNsServerPort:0], source bucket uuid=f2b5bc17672593f4309a3c5942af4df4 2024-05-31T11:52:29.481Z INFO GOXDCR.PipelineMgr: Replication Status = map[f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running} f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2:name={f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2}, status={Replicating}, errors={[]}, oldProgress={All incoming nozzles have been opened}, progress={Pipeline is running}] 2024-05-31T11:52:29.496Z INFO GOXDCR.RemClusterSvc: remoteCluster/u2jg7D8FktUa_Fp8pf_QUWGEYGTFu1-JZX55E4vXULA= useExternal: false cachedRefNodesList in refresh: [[10.0.15.212:8091 ] [10.0.9.74:8091 ]] 2024-05-31T11:52:29.564Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:30.245Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:30.619Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:31.134Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, Stats for pipeline f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2-982991818 {"Backfill Old Progress": "", "Backfill Progress": "", "CkptMgr": {"num_checkpoints": 226, "num_failedckpts": 0, "time_committing": {"count": 226, "max": 6923, "mean": 76.2433628318584, "min": 4}}, "Errors": "[]", "OldProgress": "All incoming nozzles have been opened", "Overview": {"CurrentTime": 1717156350245527164, "add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "atr_txn_docs_filtered": 0, "bandwidth_usage": 0, "binary_filtered": 0, "changes_left": 0, "client_txn_docs_filtered": 0, "data_merge_failed": 0, "data_merged": 0, "data_replicated": 491688777, "data_replicated_uncompress": 1111606573, "datapool_failed_gets": 0, "dcp_datach_length": 0, "dcp_dispatch_time": 100, "deletion_cloned": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_filtered": 0, "deletion_received_from_dcp": 0, "deletion_target_docs_skipped": 0, "docs_checked": 3174623, "docs_cloned": 0, "docs_failed_cr_source": 130, "docs_failed_cr_target": 590263, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_merge_cas_changed": 0, "docs_merge_failed": 0, "docs_merged": 0, "docs_opt_repd": 2962735, "docs_processed": 3174623, "docs_received_from_dcp": 3141094, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_unable_to_filter": 0, "docs_written": 3140964, "expiry_docs_merge_failed": 0, "expiry_docs_merged": 0, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_filtered": 0, "expiry_merge_cas_changed": 0, "expiry_received_from_dcp": 0, "expiry_stripped": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_pruned_at_merge": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "mobile_docs_filtered": 0, "num_checkpoints": 226, "num_failedckpts": 0, "pipeline_errors": 0, "pipeline_status": 1, "rate_doc_checks": 0, "rate_doc_opt_repd": 0, "rate_received_from_dcp": 0, "rate_replicated": 0, "resp_wait_time": 13, "seqno_adv_received_from_dcp": 0, "set_docs_cas_changed": 0, "set_docs_written": 3140964, "set_failed_cr_source": 130, "set_failed_cr_target": 590263, "set_filtered": 0, "set_received_from_dcp": 3141094, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "system_events_received_from_dcp": 2560, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": 0, "throughput_throttle_latency": 0, "time_committing": 76, "wtavg_docs_latency": 144, "wtavg_get_doc_latency": 0, "wtavg_merge_latency": 0, "wtavg_meta_latency": 20}, "Progress": "Pipeline is running", "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Router_dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"atr_txn_docs_filtered": 0, "binary_filtered": 0, "client_txn_docs_filtered": 0, "datapool_failed_gets": 0, "deletion_cloned": 0, "deletion_filtered": 0, "docs_cloned": 0, "docs_filtered": 0, "docs_filtered_on_txn_xattr": 0, "docs_filtered_on_user_defined_filter": 0, "docs_unable_to_filter": 0, "expiry_filtered": 0, "expiry_stripped": 0, "mobile_docs_filtered": 0, "set_filtered": 0, "throughput_throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}}, "Status": "Replicating", "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570455, "max": 42348, "mean": 56.822, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570455, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570455, "system_events_received_from_dcp": 1280}, "dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1": {"dcp_datach_length": 0, "dcp_dispatch_time": {"count": 1570639, "max": 73243, "mean": 143.902, "min": 0}, "deletion_received_from_dcp": 0, "docs_received_from_dcp": 1570639, "expiry_received_from_dcp": 0, "seqno_adv_received_from_dcp": 0, "set_received_from_dcp": 1570639, "system_events_received_from_dcp": 1280}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245781761, "data_replicated_uncompress": 555589960, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 298232, "docs_opt_repd": 1481318, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570390, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570390, "max": 557, "mean": 13.936, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570390, "set_failed_cr_source": 65, "set_failed_cr_target": 298232, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570390, "max": 790, "mean": 141.65, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 71, "mean": 20.046153846153846, "min": 0}}, "xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1": {"add_docs_cas_changed": 0, "add_docs_written": 0, "add_failed_cr_target": 0, "data_replicated": 245907016, "data_replicated_uncompress": 556016613, "datapool_failed_gets": 0, "deletion_docs_cas_changed": 0, "deletion_docs_written": 0, "deletion_failed_cr_source": 0, "deletion_failed_cr_target": 0, "deletion_target_docs_skipped": 0, "docs_failed_cr_source": 65, "docs_failed_cr_target": 292031, "docs_opt_repd": 1481417, "docs_rep_queue": 0, "docs_sent_with_subdoc_delete": 0, "docs_sent_with_subdoc_set": 0, "docs_written": 1570574, "expiry_docs_written": 0, "expiry_failed_cr_source": 0, "expiry_failed_cr_target": 0, "expiry_target_docs_skipped": 0, "guardrail_data_size": 0, "guardrail_disk_space": 0, "guardrail_resident_ratio": 0, "hlv_pruned": 0, "hlv_updated": 0, "import_docs_failed_cr_source": 0, "import_docs_written": 0, "resp_wait_time": {"count": 1570574, "max": 132, "mean": 13.547, "min": 0}, "set_docs_cas_changed": 0, "set_docs_written": 1570574, "set_failed_cr_source": 65, "set_failed_cr_target": 292031, "set_target_docs_skipped": 0, "size_rep_queue": 0, "source_sync_xattr_removed": 0, "subdoc_cmd_docs_cas_changed": 0, "target_docs_skipped": 0, "target_eaccess": 0, "target_sync_xattr_preserved": 0, "target_tmpfail": 0, "target_unknown_status": 0, "throttle_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_docs_latency": {"count": 1570574, "max": 772, "mean": 148.196, "min": 0}, "wtavg_get_doc_latency": {"count": 0, "max": 0, "mean": 0, "min": 0}, "wtavg_meta_latency": {"count": 65, "max": 82, "mean": 21.6, "min": 0}}} 2024-05-31T11:52:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_0 state =2 connType=MemConn received 1570455 items (1570455 compressed), sent 1570455 items (1570455 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 139.09847591939916ms, size of last ten batches processed [1,1,1,5,14,4,16,5,28,1], len(batches_ready_queue)=0, resend=11822, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:52:31.134Z INFO GOXDCR.XmemNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, xmem_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.15.212:11210_1 state =2 connType=MemConn received 1570639 items (1570639 compressed), sent 1570639 items (1570639 compressed), target items skipped 0, ignored 0 items, 0 items waiting to confirm, 0 in queue, 0 in current batch, avg wait time is 140.24910498211239ms, size of last ten batches processed [1,10,14,29,35,7,31,18,50,56], len(batches_ready_queue)=0, resend=12969, locked=0, repair_count_getMeta=0, repair_count_setMeta=0, retry_cr=0, to resolve=0, to setback=0, numGetMeta=65,numSubdocGet=0, temp_err=0, eaccess_err=0 guardrailHit=0 unknownStatusRec=0 2024-05-31T11:52:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_0 received 1570455 items (1570455 compressed), sent 1570455 items. 2024-05-31T11:52:31.134Z INFO GOXDCR.DcpNozzle: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, dcp_f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_10.0.2.180:11210_1 received 1570639 items (1570639 compressed), sent 1570639 items. 2024-05-31T11:52:31.135Z INFO GOXDCR.ThrSeqTrackSvc: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2_ThroughSeqnoTracker time_spent=499.533µs num_vb=512 max_sent=0 avg_sent=0 max_filtered=0 avg_filtered=0 max_failed_cr=0 avg_failed_cr=0 max_gap=0 avg_gap=0 oso_received=0 2024-05-31T11:52:31.274Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:31.479Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:32.332Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:32.495Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:33.323Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:33.508Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:34.327Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:34.398Z INFO GOXDCR.ResourceMgr: Resource Manager State = 2024-05-31T11:52:34.399Z INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=0 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0 2024-05-31T11:52:34.399Z INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[] ongoingReplMap=map[] 2024-05-31T11:52:34.602Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:35.260Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:35.470Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:36.337Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:36.354Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:36.354Z INFO GOXDCR.TopoChangeDet: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, TopologyChangeDetectorSvc for pipeline handleTargetTopologyChange completed 2024-05-31T11:52:36.501Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490 2024-05-31T11:52:37.319Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test3/test2, total_docs=3174623, docs_processed=3174623, changes_left=0 2024-05-31T11:52:37.518Z INFO GOXDCR.StatsMgr: pipelineFullTopic=f8411c7deed0a1c9dd99daeaa4e5fbde/test2/test2, total_docs=9544406, docs_processed=7272916, changes_left=2271490