Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54508

XDCR - Unable to create replications, view remote

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 7.1.3
    • 7.1.3
    • XDCR
    • Untriaged
    • 0
    • Yes

    Description

      Unable to create a replication in 7.1.3-3476 build, this last worked in 7.1.2. When I try to create a replication in this build, I get an unexpected server error, and the replication is not created. After replication creation fails, the list of remote servers does not load.

       

      Steps to reproduce:
      1. Setup 2 nodes to test XDCR.
      2. Create two buckets in source and target clusters or use the travel-sample buckets.
      3. Add target cluster as remote in source cluster for XDCR.


      4. Setup a basic replication between the source bucket and target bucket, this will return an unexpected server error and the replication creation will fail.


      5. Open XDCR page and now the list of remote clusters does not load and the spinner keeps spinning.

       

      Logs

      Source Cluster (172.23.107.65)

      http://supportal.couchbase.com/snapshot/3db922b01011f66a911c605f94c28041::0

      [s3://cb-customers-secure/mb-54508_src_cluster/2022-11-10/collectinfo-2022-11-10t102148-ns_1@cb.local.zip]

       

      Destination Cluster (172.23.107.66)

      http://supportal.couchbase.com/snapshot/237aad7541864a103cdd148fd937f683::0

      [s3://cb-customers-secure/mb-54508_dest_cluster/2022-11-10/collectinfo-2022-11-10t102211-ns_1@cb.local.zip]

       

      tail -f goxdcr*

      2022-11-10T02:18:43.890-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]

      2022-11-10T02:18:43.890-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -106

      2022-11-10T02:18:43.890-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=419 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:18:43.890-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:18:53.889-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -98

      2022-11-10T02:18:53.889-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=430 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:18:53.889-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:18:58.889-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]

      2022-11-10T02:19:03.889-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -103

      2022-11-10T02:19:03.889-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=439 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:19:03.889-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:19:13.889-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]

      2022-11-10T02:19:13.889-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -93

      2022-11-10T02:19:13.889-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=449 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:19:13.889-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:19:19.055-08:00 INFO GOXDCR.AdminPort: doCreateRemoteClusterRequest req=&

      {POST /pools/default/remoteClusters HTTP/1.1 1 1 map[Accept:[application/json, text/plain, */*] Accept-Encoding:[gzip, deflate] Accept-Language:[en-GB,en-US;q=0.9,en;q=0.8] Authorization:[Basic xxxxx] Cache-Control:[no-cache] Cb-On-Behalf-Of:[<ud>QWRtaW5pc3RyYXRvcjphZG1pbg==</ud>] Connection:[keep-alive] Content-Length:[89] Content-Type:[application/x-www-form-urlencoded; charset=UTF-8] Cookie:[ui-auth-172.23.107.65%3A8091=f735bf232c47a418f58002851d33c6a5] Invalid-Auth-Response:[on] Menelaus-Start-Time:[-576460290235] Origin:[http://172.23.107.65:8091|http://172.23.107.65:8091/] Pragma:[no-cache] Referer:[http://172.23.107.65:8091/ui/index.html] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36]] 0xc0004f7600 <nil> 89 [] false 172.23.107.65:8091 map[] map[] <nil> map[] 127.0.0.1:58956 /pools/default/remoteClusters <nil> <nil> <nil> 0xc0004f7640}

      2022-11-10T02:19:19.055-08:00 INFO GOXDCR.AdminPort: Request params: justValidate=false, remoteClusterRef=id:remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=; uuid:; name:172.23.107.66; hostName:172.23.107.66:8091; userName:<ud>Administrator</ud>; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:<nil>

      2022-11-10T02:19:19.055-08:00 INFO GOXDCR.RemClusterSvc: Adding remote cluster with referenceId remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=

      2022-11-10T02:19:19.058-08:00 INFO GOXDCR.RemClusterSvc: Result from validate remote cluster call: err=<nil>, statusCode=200. time taken=2.577926ms

      2022-11-10T02:19:19.065-08:00 INFO GOXDCR.RemoteClusterChangeListener: metakvCallback called on listener RemoteClusterChangeListener with path = /remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=

      2022-11-10T02:19:19.065-08:00 INFO GOXDCR.RemClusterSvc: metakvCallback called on path = /remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=

      2022-11-10T02:19:19.065-08:00 INFO GOXDCR.RemClusterSvc: done metakvCallback called on path = /remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=

      2022-11-10T02:19:19.066-08:00 INFO GOXDCR.RemClusterSvc: Updating remote cluster remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM= in cache after metadata store update. revision after update=[131 104 2 110 5 0 155 74 225 220 14 108 0 0 0 1 104 2 109 0 0 0 32 53 97 99 49 100 56 49 53 50 97 49 50 54 102 49 97 97 54 52 102 56 53 48 57 53 48 50 53 49 53 52 102 104 2 97 1 110 5 0 39 76 225 220 14 106]

      2022-11-10T02:19:19.066-08:00 INFO GOXDCR.RemoteClusterChangeListener: remoteClusterChangedCallback called on id = remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=, oldRef=nil, newRef=id:remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=; uuid:237aad7541864a103cdd148fd937f683; name:172.23.107.66; hostName:172.23.107.66:8091; userName:<ud>Administrator</ud>; password:xxxx; secureType:none; certificate:[]; clientCertificate:[]; clientKey:; SanInCertificate:false; HttpAuthMech:Plain, revision:[131 104 2 110 5 0 155 74 225 220 14 108 0 0 0 1 104 2 109 0 0 0 32 53 97 99 49 100 56 49 53 50 97 49 50 54 102 49 97 97 54 52 102 56 53 48 57 53 48 50 53 49 53 52 102 104 2 97 1 110 5 0 39 76 225 220 14 106]

      2022-11-10T02:19:19.066-08:00 INFO GOXDCR.RemoteClusterChangeListener: Completed remoteClusterChangedCallback called on id = remoteCluster/IieghChf16duPSNowR9Io9dcg56vLObGLqEC0T7kBAM=

      2022-11-10T02:19:19.066-08:00 INFO GOXDCR.RemClusterSvc: Agent   started for cluster:  synchronously? true

      2022-11-10T02:19:19.066-08:00 INFO GOXDCR.AdminPort: Finished doCreateRemoteClusterRequest

      2022-11-10T02:19:19.066-08:00 INFO GOXDCR.AuditSvc: GetMemcachedConnection serverAddr=127.0.0.1:11210, bucketName=

      2022-11-10T02:19:19.068-08:00 INFO GOXDCR.UILogSvc: It took 1.1e-06s to call writeUILog_async

      2022-11-10T02:19:19.068-08:00 INFO GOXDCR.AuditSvc: Successfully sent HELO command with userAgent=Goxdcr Audit. attributes={false 1 false false}

      2022-11-10T02:19:23.890-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: 58

      2022-11-10T02:19:23.890-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=459 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:19:23.890-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:19:24.260-08:00 INFO GOXDCR.AdminPort: doViewDefaultReplicationSettingsRequest

      2022-11-10T02:19:24.262-08:00 INFO GOXDCR.ReplSettSvc: Default replication settings not found in metakv

      2022-11-10T02:19:24.262-08:00 INFO GOXDCR.GlobalSettSvc: getDefaultGlobalSetting Processing = GlobalConfiguration/GlobalSettings

      2022-11-10T02:19:24.264-08:00 INFO GOXDCR.GlobalSettSvc: Global settings not found in metakv

      2022-11-10T02:19:28.498-08:00 INFO GOXDCR.AdminPort: doCreateReplicationRequest

      2022-11-10T02:19:28.499-08:00 INFO GOXDCR.AdminPort: Request parameters: justValidate=true, fromBucket=travel-sample, toCluster=172.23.107.66, toBucket=travel-sample, settings=map[CollectionsMgtMulti:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      bandwidth_limit:0 checkpoint_interval:600 compression_type:3 doc_batch_size_kb:2048 failure_restart_interval:10 filter_exp_del:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      filter_expression: filter_skip_restream:false log_level:Info optimistic_replication_threshold:256 priority:High source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500]

      2022-11-10T02:19:28.499-08:00 INFO GOXDCR.ReplMgr: Creating replication - justValidate=true, sourceBucket=travel-sample, targetCluster=172.23.107.66, targetBucket=travel-sample, settings=map[CollectionsMgtMulti:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      bandwidth_limit:0 checkpoint_interval:600 compression_type:3 doc_batch_size_kb:2048 failure_restart_interval:10 filter_exp_del:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      filter_expression: filter_skip_restream:false log_level:Info optimistic_replication_threshold:256 priority:High source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500]

      2022-11-10T02:19:28.499-08:00 INFO GOXDCR.ReplMgr: Creating replication spec - justValidate=true, sourceBucket=travel-sample, targetCluster=172.23.107.66, targetBucket=travel-sample, settings=map[CollectionsMgtMulti:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      bandwidth_limit:0 checkpoint_interval:600 compression_type:3 doc_batch_size_kb:2048 failure_restart_interval:10 filter_exp_del:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      filter_expression: filter_skip_restream:false log_level:Info optimistic_replication_threshold:256 priority:High source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500]

      2022-11-10T02:19:28.499-08:00 INFO GOXDCR.ReplSpecSvc: Start ValidateAddReplicationSpec, sourceBucket=travel-sample, targetCluster=172.23.107.66, targetBucket=travel-sample, performRPC=false settings=map[CollectionsMgtMulti:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      bandwidth_limit:0 checkpoint_interval:600 compression_type:3 doc_batch_size_kb:2048 failure_restart_interval:10 filter_exp_del:

      {map[CollectionsMgtMulti:8 filter_exp_del:0] map[collectionsExplicitMapping:false collectionsMigrationMode:false filterBypassExpiry:false filterDeletion:false filterExpiration:false]}

      filter_expression: filter_skip_restream:false log_level:Info optimistic_replication_threshold:256 priority:High source_nozzle_per_node:2 stats_interval:1000 target_nozzle_per_node:2 worker_batch_size:500]

      2022-11-10T02:19:28.499-08:00 INFO GOXDCR.ReplSpecSvc: Successfully retrieved target cluster reference 0x9818a0. time taken=24.416µs

      2022-11-10T02:19:28.500-08:00 INFO GOXDCR.ReplSpecSvc: Validated that source bucket and target bucket are not the same

      2022-11-10T02:19:28.506-08:00 INFO GOXDCR.ReplSpecSvc: Result from local bucket look up: bucketName=travel-sample, err_source=<nil>, time taken=6.828343ms

      2022-11-10T02:19:28.890-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]

      2022-11-10T02:19:32.890-08:00 ERRO GOXDCR.GenericSupervisor: Heartbeat timeout in supervisor ReplicationManagerSupervisor! not_yet_resp_count=1

      2022-11-10T02:19:32.890-08:00 INFO GOXDCR.GenericSupervisor: Child AdminportSupervisor of supervisor ReplicationManagerSupervisor missed 1 consecutive heart beats

      2022-11-10T02:19:32.891-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:33.890-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -107

      2022-11-10T02:19:33.890-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=469 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:19:33.890-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:19:34.889-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:37.889-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:40.890-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:43.890-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -107

      2022-11-10T02:19:43.890-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=479 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:19:43.890-08:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":5736736,"TotalAlloc":10330864,"Sys":20005896,"Lookups":0,"Mallocs":77230,"Frees":51693,"HeapAlloc":5736736,"HeapSys":11599872,"HeapIdle":4202496,"HeapInuse":7397376,"HeapReleased":3948544,"HeapObjects":25537,"StackInuse":983040,"StackSys":983040,"MSpanInuse":100912,"MSpanSys":114240,"MCacheInuse":4800,"MCacheSys":15600,"BuckHashSys":1450969,"GCSys":5248560,"OtherSys":593615,"NextGC":9320992,"LastGC":1668075463991678094,"PauseTotalNs":378210,"PauseNs":[45519,101854,59478,97846,73513,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[1668075103747185070,1668075103755434422,1668075223790630202,1668075343891181774,1668075463991678094,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":5,"NumForcedGC":0,"GCCPUFraction":0.000005570857741877423,"EnableGC":true,"DebugGC":false,"BySize":[

      {"Size":0,"Mallocs":0,"Frees":0}

      ,{"Size":8,"Mallocs":862,"Frees":467},{"Size":16,"Mallocs":21904,"Frees":13232},{"Size":24,"Mallocs":7982,"Frees":4513},{"Size":32,"Mallocs":4492,"Frees":3303},{"Size":48,"Mallocs":13798,"Frees":9586},{"Size":64,"Mallocs":4952,"Frees":3654},{"Size":80,"Mallocs":2181,"Frees":1310},{"Size":96,"Mallocs":4421,"Frees":2838},{"Size":112,"Mallocs":2617,"Frees":2012},{"Size":128,"Mallocs":1117,"Frees":876},{"Size":144,"Mallocs":1211,"Frees":857},{"Size":160,"Mallocs":434,"Frees":248},{"Size":176,"Mallocs":167,"Frees":104},{"Size":192,"Mallocs":165,"Frees":81},{"Size":208,"Mallocs":559,"Frees":373},{"Size":224,"Mallocs":437,"Frees":327},{"Size":240,"Mallocs":32,"Frees":12},{"Size":256,"Mallocs":1139,"Frees":924},{"Size":288,"Mallocs":805,"Frees":513},{"Size":320,"Mallocs":169,"Frees":116},{"Size":352,"Mallocs":1499,"Frees":1080},{"Size":384,"Mallocs":314,"Frees":211},{"Size":416,"Mallocs":185,"Frees":23},{"Size":448,"Mallocs":4,"Frees":0},{"Size":480,"Mallocs":10,"Frees":4},{"Size":512,"Mallocs":164,"Frees":111},{"Size":576,"Mallocs":114,"Frees":49},{"Size":640,"Mallocs":242,"Frees":39},{"Size":704,"Mallocs":127,"Frees":76},{"Size":768,"Mallocs":9,"Frees":6},{"Size":896,"Mallocs":52,"Frees":28},{"Size":1024,"Mallocs":46,"Frees":20},{"Size":1152,"Mallocs":61,"Frees":29},{"Size":1280,"Mallocs":597,"Frees":577},{"Size":1408,"Mallocs":50,"Frees":17},{"Size":1536,"Mallocs":6,"Frees":3},{"Size":1792,"Mallocs":23,"Frees":5},{"Size":2048,"Mallocs":29,"Frees":10},{"Size":2304,"Mallocs":40,"Frees":18},{"Size":2688,"Mallocs":29,"Frees":16},{"Size":3072,"Mallocs":15,"Frees":7},{"Size":3200,"Mallocs":1,"Frees":0},{"Size":3456,"Mallocs":3,"Frees":2},{"Size":4096,"Mallocs":90,"Frees":25},{"Size":4864,"Mallocs":10,"Frees":3},{"Size":5376,"Mallocs":16,"Frees":9},{"Size":6144,"Mallocs":6,"Frees":1},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":1,"Frees":1},{"Size":6912,"Mallocs":1,"Frees":0},{"Size":8192,"Mallocs":12,"Frees":2},{"Size":9472,"Mallocs":10,"Frees":4},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":5,"Frees":2},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":2,"Frees":0},{"Size":14336,"Mallocs":24,"Frees":0},{"Size":16384,"Mallocs":4,"Frees":1},{"Size":18432,"Mallocs":4,"Frees":1}]}

      2022-11-10T02:19:43.890-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

      2022-11-10T02:19:43.890-08:00 INFO GOXDCR.PipelineMgr: Replication Status = map[]

      2022-11-10T02:19:43.890-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:46.889-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:49.890-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:52.889-08:00 INFO GOXDCR.GenericSupervisor: Send heartbeat failed for AdminportSupervisor, err=Last heart beat msg has not been processed

      2022-11-10T02:19:53.889-08:00 INFO GOXDCR.ResourceMgr: Resource Manager State = overallTP: 0 highTP: 0 highExist: false lowExist: false backlogExist: false maxTP: 0 highTPNeeded: 0 highTokens: 0 maxTokens: 0 lowTPLimit: 0 calibration: None dcpAction: Reset processCpu: 0 idleCpu: -123

      2022-11-10T02:19:53.889-08:00 INFO GOXDCR.ResourceMgr: backlogCount=0, noBacklogCount=489 extraQuota=false cpuNotMaxedCount=0 throughputDropCount=0

      2022-11-10T02:19:53.889-08:00 INFO GOXDCR.ResourceMgr: DcpPriorityMap=map[]

      ongoingReplMap=map[]

       

      tail -f goxdcr | grep ERRO*

      2022-11-10T02:20:52.790-08:00 ERRO GOXDCR.HttpServer: Execution timed out

      2022-11-10T02:20:55.246-08:00 ERRO GOXDCR.HttpServer: Execution timed out

      2022-11-10T02:21:00.046-08:00 ERRO GOXDCR.HttpServer: Execution timed out

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-54416 [BP 7.1.X] XDCR - AdvFilter upgrade happens pre-emptively leading to missed documents

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          mentioned in

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ayush.nayyar Ayush Nayyar
              ayush.nayyar Ayush Nayyar
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty