Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30362

very high XDCR replication lag over WAN

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 5.5.0
    • 5.5.0
    • XDCR
    • Untriaged
    • Yes

    Description

      Replication lag is 100+ sec since build  5.5.0-2935:  http://showfast.sc.couchbase.com/#/timeline/Linux/xdcr/ongoing/all

       

      The 2935 was about updating gocb and gocbcore:

      http://172.23.120.24/builds/latestbuilds/couchbase-server/vulcan/2935/CHANGELOG

         godeps/src/github.com/couchbase/gocb
         godeps/src/gopkg.in/couchbase/gocbcore.v7

       

      Environment:

      Cluster: titan_5x5 
      OS: CentOS 7 
      CPU: E5-2680 v3 (48 vCPU) 
      Memory: 256 GB 
      Disk: Samsung PM863a 

      Test:

      95th percentile replication lag
      5 -> 5 (2 source nozzles, 4 target nozzles)
      1 bucket, 1B x 1KB items,  40K updates/sec ongoing 
      WAN 80±4 ms

       

      Logs from the run on 5.5.0-2935:
      Source:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.105.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.106.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.107.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.108.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.109.zip

      Destination:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.100.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.101.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.102.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.103.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.104.zip

       

       

      Comparison between 5.5.0-2934 and 2935:

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_550-2935_access_0105&label=5.5.0-2935&snapshot=titan_c1_550-2934_access_16c6&label=5.5.0-2934

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c2_550-2935_access_df73&label=5.5.0-2935&snapshot=titan_c2_550-2934_access_37ae&label=5.5.0-2934

       

       

       

       

       

       

       

      Attachments

        Activity

          People

            neil.huang Neil Huang
            oleksandr.gyryk Alex Gyryk (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty