Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43734

99.9th percentile GET/SET Latency of magma single-node latency tests increased on build 7.0.0-4226

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma

       

      Test: 1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma

      Build Latency Job
      7.0.0-4170 17.17 (GET)
      7.59 (SET)
      http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/ 
      7.0.0-4226 25.98 (GET)
      18.29 (SET)
      http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/17/ 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          • * Commit: ec04461418ae981f500e6575809c5b6fedba0bcb in build: couchbase-server-7.0.0-4212
            CBSS-501 magma: Fix and enable file histogram stats
          sarath Sarath Lakshman added a comment - * Commit: ec04461418ae981f500e6575809c5b6fedba0bcb in build: couchbase-server-7.0.0-4212 CBSS-501 magma: Fix and enable file histogram stats
          apaar.gupta Apaar Gupta added a comment -

          Bo-Chun Wang the regression should be fixed in build 4250. Could you rerun the tests?

          apaar.gupta Apaar Gupta added a comment - Bo-Chun Wang the regression should be fixed in build 4250. Could you rerun the tests?
          bo-chun.wang Bo-Chun Wang added a comment -

          I finished runs on build 4250. The regression still exists.

           

          Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma

           

          Test: 1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma

          Build Latency Job
          7.0.0-4170 17.17 (GET)
          7.59 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/ 
          7.0.0-4226 25.98 (GET)
          18.29 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/17/ 
          7.0.0-4250 18.85 (GET)
          12.21 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/27/ 
          bo-chun.wang Bo-Chun Wang added a comment - I finished runs on build 4250. The regression still exists.   Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma Build Latency Job 7.0.0-4170 6.16 (GET) 4.7 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/9/ 7.0.0-4199 6.77 (GET) 4.06 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/22/ 7.0.0-4212 21.47 (GET) 14.19 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/12/ 7.0.0-4226 23.16 (GET) 18.01 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/18/ 7.0.0-4250 24.77 (GET) 19.84 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/26/     Test:   1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma Build Latency Job 7.0.0-4170 17.17 (GET) 7.59 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/   7.0.0-4226 25.98 (GET) 18.29 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/17/   7.0.0-4250 18.85 (GET) 12.21 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/27/  
          apaar.gupta Apaar Gupta added a comment -

          Pushed a fix disabling the histograms causing the issue. Verified it by running few of the tests. Leaving this ticket open to track debugging and re-enabling the histograms.

          apaar.gupta Apaar Gupta added a comment - Pushed a fix disabling the histograms causing the issue. Verified it by running few of the tests. Leaving this ticket open to track debugging and re-enabling the histograms.
          bo-chun.wang Bo-Chun Wang added a comment - - edited

          The latency of 15K test is back to normal. The set latency of 50K test is still higher.

           

          Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma

           

          Test: 1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma

          Build Latency Job
          7.0.0-4170 17.17 (GET)
          7.59 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/ 
          7.0.0-4226 25.98 (GET)
          18.29 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/17/ 
          7.0.0-4263 22.37 (GET)
          14.84 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/33/ 
          bo-chun.wang Bo-Chun Wang added a comment - - edited The latency of 15K test is back to normal. The set latency of 50K test is still higher.   Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma Build Latency Job 7.0.0-4170 6.16 (GET) 4.7 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/9/ 7.0.0-4226 23.16 (GET) 18.01 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/18/ 7.0.0-4263 4.41 (GET) 2.33 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/32/     Test:   1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma Build Latency Job 7.0.0-4170 17.17 (GET) 7.59 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/   7.0.0-4226 25.98 (GET) 18.29 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/17/   7.0.0-4263 22.37 (GET) 14.84 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/33/  
          lynn.straus Lynn Straus added a comment -

          Per review in the Cheshire Cat Ops meeting and per email updates, all remaining majors/minors/trivials are being moved out of Cheshire Cat on March 10. The purpose is to allow teams to focus on Critical bugs for the remainder of the release. Teams can "pull back" deferred bugs as schedule and bandwidth allow.

          Changing target fix version to CheshireCat.next and adding "deferred-from-Cheshire-Cat" label.

          lynn.straus Lynn Straus added a comment - Per review in the Cheshire Cat Ops meeting and per email updates, all remaining majors/minors/trivials are being moved out of Cheshire Cat on March 10. The purpose is to allow teams to focus on Critical bugs for the remainder of the release. Teams can "pull back" deferred bugs as schedule and bandwidth allow. Changing target fix version to CheshireCat.next and adding "deferred-from-Cheshire-Cat" label.

          Closing this ticket as the latency numbers have significantly improved in 7.0.0-4777.

          sarath Sarath Lakshman added a comment - Closing this ticket as the latency numbers have significantly improved in 7.0.0-4777.

          I close this issue. The performance regression is fixed.

          Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma

           

          Test: 1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma

          Build Latency Job
          7.0.0-4170 17.17 (GET)
          7.59 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/ 
          7.0.0-4797 11.35 (GET)
          3.53 (SET)
          http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/88/ 

           

          bo-chun.wang Bo-Chun Wang added a comment - I close this issue. The performance regression is fixed. Test: 1 nodes, 100M x 1KB, 15K ops/sec, 50/50 R/W, Uniform distribution, Magma Build Latency Job 7.0.0-4170 6.16 (GET) 4.7 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/9/ 7.0.0-4797 5.46 (GET) 2.6 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/89/   Test:   1 nodes, 100M x 1KB, 50K ops/sec, 50/50 R/W, Uniform distribution, Magma Build Latency Job 7.0.0-4170 17.17 (GET) 7.59 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/5/   7.0.0-4797 11.35 (GET) 3.53 (SET) http://perf.jenkins.couchbase.com/job/magma-nvme-1node-latency/88/    

          People

            apaar.gupta Apaar Gupta
            bo-chun.wang Bo-Chun Wang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty