Details
-
Bug
-
Resolution: Fixed
-
Major
-
Morpheus, 7.6.2
-
Untriaged
-
0
-
Unknown
Description
When running with mobile enabled and sgw connected to the source cluster, XDCR replicates almost all documents, then gets stuck at xdcr_changes_left_total = 1.
This is because sgw has a heartbeat doc that is written every second, which increments the total_docs number, so replication will not complete:
2024-02-13T07:07:29.050-08:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=9d4ee403636db763b3865cc32cee5a45/bucket-1/bucket-1, total_docs=21012393, docs_processed=21012392, changes_left=1
|
2024-02-13T07:07:30.043-08:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=9d4ee403636db763b3865cc32cee5a45/bucket-1/bucket-1, total_docs=21012394, docs_processed=21012393, changes_left=1
|
2024-02-13T07:07:31.043-08:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=9d4ee403636db763b3865cc32cee5a45/bucket-1/bucket-1, total_docs=21012395, docs_processed=21012394, changes_left=1
|
2024-02-13T07:07:32.036-08:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=9d4ee403636db763b3865cc32cee5a45/bucket-1/bucket-1, total_docs=21012396, docs_processed=21012395, changes_left=1
|
2024-02-13T07:07:33.044-08:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=9d4ee403636db763b3865cc32cee5a45/bucket-1/bucket-1, total_docs=21012397, docs_processed=21012396, changes_left=1
|
2024-02-13T07:07:34.050-08:00 INFO GOXDCR.StatsMgr: pipelineFullTopic=9d4ee403636db763b3865cc32cee5a45/bucket-1/bucket-1, total_docs=21012398, docs_processed=21012397, changes_left=1
|
Cbcollect logs: https://s3-us-west-2.amazonaws.com/perf-artifacts//172.23.100.190.zip
The heartbeat docs looks something like this:
{
|
"$1": {
|
"cas": 1707846795786977300,
|
"expiration": 0,
|
"flags": 50331648,
|
"id": "_sync:heartbeat_timeout:530505cf09cb84a4",
|
"keyspace": "default:bucket-1",
|
"type": "base64"
|
}
|
},
|
]
|
Mobile mode seems to be correctly set to active, based on the output of this curl command:
curl http://172.23.100.190:8091/settings/replications/00d59c0dfd1d07a2c0f428a2eb8783aa%2Fbucket-1%2Fbucket-1
|
{"checkpointInterval":600,"ckptSvcCacheEnabled":true,"colMappingRules":{},"collectionsExplicitMapping":false,"collectionsMigrationMode":false,"collectionsMirroringMode":false,"collectionsOSOMode":true,"compressionType":"Auto","dcpEnablePurgeRollback":false,"desiredLatency":50,"docBatchSizeKb":2048,"failureRestartInterval":10,"filterBinary":false,"filterBypassExpiry":false,"filterBypassUncommittedTxn":false,"filterDeletion":false,"filterExpiration":false,"filterExpression":"","jsFunctionTimeoutMs":20000,"logLevel":"Info","mergeFunctionMapping":{},"mobile":"Active","networkUsageLimit":0,"optimisticReplicationThreshold":256,"pauseRequested":false,"preReplicateVBMasterCheck":true,"priority":"High","replicateCkptIntervalMin":20,"retryOnErrExceptAuthErrMaxWaitSec":360,"retryOnRemoteAuthErr":true,"retryOnRemoteAuthErrMaxWaitSec":360,"sourceNozzlePerNode":2,"statsInterval":1000,"targetNozzlePerNode":4,"type":"xmem","workerBatchSize":500}
|