Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24565

[XDCR] bandwidth limit is reset to 0 when BandwidthThrottlerSvc is attached to pipeline - user setting through rest call overridden

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      5.0.0-2915 - Steps followed:

      1. Create 1x1 unixdcr:

      2017-05-26T00:28:30.659-07:00 INFO GOXDCR.AdminPort: doCreateReplicationRequest
      		 
      2017-05-26T00:28:30.660-07:00 INFO GOXDCR.AdminPort: Request parameters: justValidate=false, fromBucket=default, toCluster=remote_cluster_C1-C2, toBucket=default, settings=map[replication_type:xmem]
      		 
      2017-05-26T00:28:30.660-07:00 INFO GOXDCR.ReplMgr: Creating replication - justValidate=false, sourceBucket=default, targetCluster=remote_cluster_C1-C2, targetBucket=default, settings=map[replication_type:xmem]
      		 
      2017-05-26T00:28:30.660-07:00 INFO GOXDCR.ReplMgr: Creating replication spec - justValidate=false, sourceBucket=default, targetCluster=remote_cluster_C1-C2, targetBucket=default, settings=map[replication_type:xmem]
      		 
      2017-05-26T00:28:30.660-07:00 INFO GOXDCR.ReplSpecSvc: Start ValidateAddReplicationSpec, sourceBucket=default, targetCluster=remote_cluster_C1-C2, targetBucket=default
      		 
      2017-05-26T00:28:30.674-07:00 INFO GOXDCR.ReplSpecSvc: Result from local bucket look up: bucketName=default, err_source=<nil>, time taken=13.808477ms
      		 
      2017-05-26T00:28:30.677-07:00 INFO GOXDCR.ReplSpecSvc: Successfully retrieved target cluster reference remote_cluster_C1-C2. time take=2.411672ms
      		 
      2017-05-26T00:28:30.678-07:00 INFO GOXDCR.ReplSpecSvc: Validated that source bucket and target bucket are not the same. time taken=1.332786ms
      		 
      2017-05-26T00:28:30.691-07:00 INFO GOXDCR.ReplSpecSvc: Result from remote bucket look up: connStr=172.23.107.161:8091, bucketName=default, err_target=<nil>, time taken=13.295658ms
      		 
      2017-05-26T00:28:30.692-07:00 INFO GOXDCR.ReplSpecSvc: Finished ValidateAddReplicationSpec. errorMap=map[]
      		 
      2017-05-26T00:28:30.699-07:00 INFO GOXDCR.ReplSpecSvc: Start AddReplicationSpec, spec=&{659a6f685b39519d7dca3794d4e1faf7/default/default default aa549cad60238d3683a094de2e9c584a 659a6f685b39519d7dca3794d4e1faf7 default 75ac259541f003731b41292a17ba634b 0xc4201e8e60 <nil>}
      		 
      2017-05-26T00:28:30.699-07:00 INFO GOXDCR.ReplSpecSvc: Adding it to metadata store...
      		 
      2017-05-26T00:28:30.702-07:00 INFO GOXDCR.ReplicationSpecChangeListener: specChangedCallback called on id = 659a6f685b39519d7dca3794d4e1faf7/default/default, oldSpec=<nil>, newSpec=&{659a6f685b39519d7dca3794d4e1faf7/default/default default aa549cad60238d3683a094de2e9c584a 659a6f685b39519d7dca3794d4e1faf7 default 75ac259541f003731b41292a17ba634b 0xc4201e8e60 <nil>}
      		 
      2017-05-26T00:28:30.702-07:00 INFO GOXDCR.ReplicationSpecChangeListener: new spec settings=&{xmem  true 1800 500 2048 10 256 2 2 1000 50 Info 1000 0 [131 108 0 0 0 1 104 2 109 0 0 0 32 50 97 101 102 53 98 55 102 56 49 49 53 52 97 97 55 48 54 97 56 98 98 51 57 54 101 97 99 55 51 54 97 104 2 97 1 110 5 0 30 85 156 210 14 106]}
      		 
      2017-05-26T00:28:30.702-07:00 INFO GOXDCR.ReplicationSpecChangeListener: Starting pipeline 659a6f685b39519d7dca3794d4e1faf7/default/default since the replication spec has been changed to active
      		 
      2017-05-26T00:28:30.702-07:00 INFO GOXDCR.ReplMgr: Success adding replication specification 659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.702-07:00 INFO GOXDCR.ReplMgr: Replication specification 659a6f685b39519d7dca3794d4e1faf7/default/default is created
      		 
      2017-05-26T00:28:30.702-07:00 INFO GOXDCR.AdminPort: Finished doCreateReplicationRequest call

      2. Set bandwidth_limit = 1 Mb/s through rest call:

      2017-05-26T00:28:30.745-07:00 INFO GOXDCR.AdminPort: doChangeReplicationSettingsRequest
      		 
      2017-05-26T00:28:30.745-07:00 INFO GOXDCR.AdminPort: Request params: replicationId=659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.745-07:00 INFO GOXDCR.AdminPort: Request params: justValidate=false, inputSettings=map[bandwidth_limit:1]
      		 
      2017-05-26T00:28:30.745-07:00 INFO GOXDCR.ReplMgr: Update replication settings for 659a6f685b39519d7dca3794d4e1faf7/default/default, settings=map[bandwidth_limit:1]
      		 
      2017-05-26T00:28:30.747-07:00 INFO GOXDCR.ReplicationSpecChangeListener: metakvCallback called on listener ReplicationSpecChangeListener with path = /replicationSpec/659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.747-07:00 INFO GOXDCR.ReplSpecSvc: ReplicationSpecServiceCallback called on path = /replicationSpec/659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.747-07:00 INFO GOXDCR.ReplicationSpecChangeListener: specChangedCallback called on id = 659a6f685b39519d7dca3794d4e1faf7/default/default, oldSpec=&{659a6f685b39519d7dca3794d4e1faf7/default/default default aa549cad60238d3683a094de2e9c584a 659a6f685b39519d7dca3794d4e1faf7 default 75ac259541f003731b41292a17ba634b 0xc4201e9360 [131 108 0 0 0 1 104 2 109 0 0 0 32 50 97 101 102 53 98 55 102 56 49 49 53 52 97 97 55 48 54 97 56 98 98 51 57 54 101 97 99 55 51 54 97 104 2 97 2 110 5 0 30 85 156 210 14 106]}, newSpec=&{659a6f685b39519d7dca3794d4e1faf7/default/default default aa549cad60238d3683a094de2e9c584a 659a6f685b39519d7dca3794d4e1faf7 default 75ac259541f003731b41292a17ba634b 0xc4201e9c20 [131 108 0 0 0 1 104 2 109 0 0 0 32 50 97 101 102 53 98 55 102 56 49 49 53 52 97 97 55 48 54 97 56 98 98 51 57 54 101 97 99 55 51 54 97 104 2 97 3 110 5 0 30 85 156 210 14 106]}
      		 
      2017-05-26T00:28:30.747-07:00 INFO GOXDCR.ReplicationSpecChangeListener: old spec settings=&{xmem  true 60 500 2048 10 256 2 2 0 0 Info 1000 0 <nil>}
      		 
      2017-05-26T00:28:30.747-07:00 INFO GOXDCR.ReplicationSpecChangeListener: new spec settings=&{xmem  true 60 500 2048 10 256 2 2 0 0 Info 1000 1 <nil>}
      		 
      2017-05-26T00:28:30.747-07:00 INFO GOXDCR.ReplicationSpecChangeListener: Updating pipeline 659a6f685b39519d7dca3794d4e1faf7/default/default with new settings=&{xmem  true 60 500 2048 10 256 2 2 0 0 Info 1000 1 <nil>}
      		 
       old settings=&{xmem  true 60 500 2048 10 256 2 2 0 0 Info 1000 0 <nil>}
      		 
      2017-05-26T00:28:30.747-07:00 ERRO GOXDCR.ReplicationSpecChangeListener: Failed to perform live update on pipeline 659a6f685b39519d7dca3794d4e1faf7/default/default, err=Cannot find pipeline with topic 659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.747-07:00 ERRO GOXDCR.ReplSpecSvc: Cannot find pipeline with topic 659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.751-07:00 INFO GOXDCR.ReplSpecSvc: Replication spec 659a6f685b39519d7dca3794d4e1faf7/default/default has been updated, rev=[131 108 0 0 0 1 104 2 109 0 0 0 32 50 97 101 102 53 98 55 102 56 49 49 53 52 97 97 55 48 54 97 56 98 98 51 57 54 101 97 99 55 51 54 97 104 2 97 3 110 5 0 30 85 156 210 14 106]
      		 
      2017-05-26T00:28:30.751-07:00 INFO GOXDCR.ReplMgr: Updated replication settings for replication 659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.751-07:00 INFO GOXDCR.ReplMgr: Done with replication settings auditing for replication 659a6f685b39519d7dca3794d4e1faf7/default/default
      		 
      2017-05-26T00:28:30.751-07:00 INFO GOXDCR.AdminPort: Done with doChangeReplicationSettingsRequest

      3. BandwidthThrottlerSvc is attached to pipeline - resetting bandwidth limit to 0:

      2017-05-26T00:28:30.802-07:00 INFO GOXDCR.RuntimeCtx: Try to attach BandwidthThrottlerSvc to pipeline 659a6f685b39519d7dca3794d4e1faf7/default/default-773061494
      		 
      2017-05-26T00:28:30.805-07:00 INFO GOXDCR.BwThrottler: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc set overall bandwidth limit to 0 and number of source nodes to 1
      		 
      2017-05-26T00:28:30.805-07:00 INFO GOXDCR.BwThrottler: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc updated bandwidth limit to 0
      		 
      2017-05-26T00:28:30.805-07:00 INFO GOXDCR.BwThrottler: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc attached to pipeline
      		 
      2017-05-26T00:28:30.805-07:00 INFO GOXDCR.XDCRFactory: Pipeline 659a6f685b39519d7dca3794d4e1faf7/default/default has been constructed

      4. As a result bandwidth_limit and bandwidth_usage are always 0:

      [root@localhost logs]#  zgrep "bandwidth_limit=" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log*
      		 
      shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
      		 
      2017-05-26T00:28:40.864-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:28:50.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:29:00.847-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:29:10.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:29:20.855-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:29:30.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:29:40.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:29:50.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:30:00.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:30:10.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:30:20.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:30:30.853-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:30:40.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:30:50.850-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:31:00.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:31:10.859-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:31:20.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:31:30.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:31:40.847-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:31:50.849-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:32:00.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:32:10.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:32:20.847-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:32:30.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:32:40.853-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:32:50.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:33:00.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:33:10.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:33:20.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:33:30.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:33:40.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:33:50.849-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:34:00.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:34:10.849-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:34:20.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:34:30.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:34:40.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:34:50.848-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:35:00.846-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0
      		 
      2017-05-26T00:35:10.850-07:00 INFO GOXDCR.StatsMgr: 659a6f685b39519d7dca3794d4e1faf7/default/default_BandwidthThrottlerSvc bandwidth_limit=0, bandwidth_usage=0

      5. UI reflects bandwidth usage as 1 MB/s - please see screenshot

      6. Rest GET shows it as 0:

      Arunkumars-MBP:arun-testrunner arunkumarsenthilnathan$ curl -u Administrator:password -X GET http://172.23.106.251:8091/settings/replications

      {"checkpointInterval":1800,"docBatchSizeKb":2048,"failureRestartInterval":10,"goGC":100,"goMaxProcs":4,"logLevel":"Info","networkUsageLimit":0,"optimisticReplicationThreshold":256,"sourceNozzlePerNode":2,"statsInterval":1000,"targetNozzlePerNode":2,"workerBatchSize":500}

      Arunkumars-MBP:arun-testrunner arunkumarsenthilnathan$

      Logs: https://s3.amazonaws.com/bugdb/jira/may26/collectinfo-2017-05-26T075333-ns_1%40127.0.0.1.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            arunkumar Arunkumar Senthilnathan (Inactive)
            arunkumar Arunkumar Senthilnathan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty