Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30362

very high XDCR replication lag over WAN

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Cannot Reproduce
    • 5.5.0
    • 5.5.0
    • XDCR
    • Untriaged
    • Yes

    Description

      Replication lag is 100+ sec since build  5.5.0-2935:  http://showfast.sc.couchbase.com/#/timeline/Linux/xdcr/ongoing/all

       

      The 2935 was about updating gocb and gocbcore:

      http://172.23.120.24/builds/latestbuilds/couchbase-server/vulcan/2935/CHANGELOG

         godeps/src/github.com/couchbase/gocb
         godeps/src/gopkg.in/couchbase/gocbcore.v7

       

      Environment:

      Cluster: titan_5x5 
      OS: CentOS 7 
      CPU: E5-2680 v3 (48 vCPU) 
      Memory: 256 GB 
      Disk: Samsung PM863a 

      Test:

      95th percentile replication lag
      5 -> 5 (2 source nozzles, 4 target nozzles)
      1 bucket, 1B x 1KB items,  40K updates/sec ongoing 
      WAN 80±4 ms

       

      Logs from the run on 5.5.0-2935:
      Source:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.105.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.106.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.107.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.108.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.109.zip

      Destination:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.100.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.101.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.102.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.103.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-3426/172.23.96.104.zip

       

       

      Comparison between 5.5.0-2934 and 2935:

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_550-2935_access_0105&label=5.5.0-2935&snapshot=titan_c1_550-2934_access_16c6&label=5.5.0-2934

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c2_550-2935_access_df73&label=5.5.0-2935&snapshot=titan_c2_550-2934_access_37ae&label=5.5.0-2934

       

       

       

       

       

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          I'm still investigating this issue, will post my findings if have some

          oleksandr.gyryk Alex Gyryk (Inactive) added a comment - I'm still investigating this issue, will post my findings if have some
          jliang John Liang added a comment -

          Alex, goxdcr is not using gocb.

          jliang John Liang added a comment - Alex, goxdcr is not using gocb.

          Interesting. There is no other changes in 2935 and 2934 is good. Ok, I'll schedule few more runs on other builds to make sure its 2935 

          oleksandr.gyryk Alex Gyryk (Inactive) added a comment - Interesting. There is no other changes in 2935 and 2934 is good. Ok, I'll schedule few more runs on other builds to make sure its 2935 

          the number is back to normal with the latest RC3 build.

          probably environmental issue so I'm closing the ticket

          oleksandr.gyryk Alex Gyryk (Inactive) added a comment - the number is back to normal with the latest RC3 build. probably environmental issue so I'm closing the ticket

          People

            neil.huang Neil Huang
            oleksandr.gyryk Alex Gyryk (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty