Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28896

XDCR throughput drop after adding scram-sha support

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 5.5.0
    • 5.5.0
    • XDCR
    • Cluster: titan_5x5
      OS: CentOS 7
      CPU: E5-2680 v3 (48 vCPU)
      Memory: 256 GB
      Disk: Samsung PM863a

    Description

      XDCR rate degraded from 160K to 120K items/sec 
      (all results: http://showfast.sc.couchbase.com/#/timeline/Linux/xdcr/init_single/all)

      XDCR configuration: 
      1 to 1 replication
      sourceNozzlePerNode = 2
      targetNozzlePerNode = 4

       

      Detected on build 5.5.0-2167

      goxdcr changes in 2267: https://github.com/couchbase/goxdcr/commit/1a384835487379c32e6714aa7e4097253534b35b

       

      5.5.0-2166 and 5.5.0-2167 builds comparison:
      source: [http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_550-2166_init_xdcr_ebce&snapshot=titan_c1_550-2167_init_xdcr_e4dd
      d|http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c1_550-2166_init_xdcr_ebce&snapshot=titan_c1_550-2167_init_xdcr_e4dd]estination: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=titan_c2_550-2166_init_xdcr_7973&snapshot=titan_c2_550-2167_init_xdcr_96a2

       

      Logs from 5.5.0-2167 run:

      source: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-2836/172.23.96.105.zip
      destination: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-2836/172.23.96.100.zip

       

       

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Alex Gyryk I went over the scram sha checkin a couple times and could not figure out how it could lead to the observed performance degradation. I built a toy build below, which is simply adding the scram sha checkin on top of 2166 build. Can you please re-run the test against this toy build? If it shows the same performance issue, the scram sha checkin is definitely the culprit.

          http://server.jenkins.couchbase.com/job/toy-unix/2444/

          yu Yu Sui (Inactive) added a comment - Alex Gyryk I went over the scram sha checkin a couple times and could not figure out how it could lead to the observed performance degradation. I built a toy build below, which is simply adding the scram sha checkin on top of 2166 build. Can you please re-run the test against this toy build? If it shows the same performance issue, the scram sha checkin is definitely the culprit. http://server.jenkins.couchbase.com/job/toy-unix/2444/
          pavelpaulau Pavel Paulau (Inactive) added a comment - https://hub.internal.couchbase.com/confluence/display/PERF/Running+performance+tests+for+a+toy+build

          Thanks for the info. Will try.

          yu Yu Sui (Inactive) added a comment - Thanks for the info. Will try.

          Build couchbase-server-5.5.0-2570 contains goxdcr commit 4f1344fd04389d5a0db3b71934da8b4f3e9357d9 with commit message:
          MB-28896 randomize vb streams to start to ensure even load across out nozzles
          https://github.com/couchbase/goxdcr/commit/4f1344fd04389d5a0db3b71934da8b4f3e9357d9

          build-team Couchbase Build Team added a comment - Build couchbase-server-5.5.0-2570 contains goxdcr commit 4f1344fd04389d5a0db3b71934da8b4f3e9357d9 with commit message: MB-28896 randomize vb streams to start to ensure even load across out nozzles https://github.com/couchbase/goxdcr/commit/4f1344fd04389d5a0db3b71934da8b4f3e9357d9

          Build couchbase-server-6.0.0-1053 contains goxdcr commit 4f1344fd04389d5a0db3b71934da8b4f3e9357d9 with commit message:
          MB-28896 randomize vb streams to start to ensure even load across out nozzles
          https://github.com/couchbase/goxdcr/commit/4f1344fd04389d5a0db3b71934da8b4f3e9357d9

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.0-1053 contains goxdcr commit 4f1344fd04389d5a0db3b71934da8b4f3e9357d9 with commit message: MB-28896 randomize vb streams to start to ensure even load across out nozzles https://github.com/couchbase/goxdcr/commit/4f1344fd04389d5a0db3b71934da8b4f3e9357d9

          People

            oleksandr.gyryk Alex Gyryk (Inactive)
            oleksandr.gyryk Alex Gyryk (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty