Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39856

Tools/Export performance dropped on 7.0.0-2278 and 6.6.0-7737

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: Cheshire-Cat, 6.6.0
    • Fix Version/s: None
    • Component/s: test-execution
    • Story Points:
      1

      Description

      Weekly performance runs show that Tools/Export performance has dropped on 7.0.0-2278 and 6.6.0-7737.

      Export JSON Lines:

       

       

      Build

      Avg. MB/sec

      Job Diff
      7.0.0-2147 156 http://perf.jenkins.couchbase.com/job/leto/13019/  
      7.0.0-2278 128 http://perf.jenkins.couchbase.com/job/leto/13309/ -18%

       

      Build

      Avg. MB/sec

      Job Diff
      6.6.0-7692 128 http://perf.jenkins.couchbase.com/job/leto/13075/  
      6.6.0-7737 115 http://perf.jenkins.couchbase.com/job/leto/13202/ -10%

       

      Export JSON List: 

      Build

      Avg. MB/sec

      Job Diff
      7.0.0-2147 165 http://perf.jenkins.couchbase.com/job/leto/13021/  
      7.0.0-2278 131 http://perf.jenkins.couchbase.com/job/leto/13311/ -20%

       

      Build

      Avg. MB/sec

      Job Diff
      6.6.0-7692 122 http://perf.jenkins.couchbase.com/job/leto/13076/  
      6.6.0-7737 100 http://perf.jenkins.couchbase.com/job/leto/13203/ -18%

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          james.lee James Lee added a comment -

          Hi Bo-Chun Wang,

          I've been having a little look into why this might be happening because there haven't been many (if any) major code changes to cbexport between those two builds; the only thing I can think of is that by default we will now request
          that Couchbase Server sequentially backfills the vBuckets. I have my doubts whether this may be the cause of the issue because if we look at the latest backup to blackhole test we can see that the throughput has increased (backup/export share the same code in this respect).

          I did some local testing of cbexport (which should be taken with a grain of salt due to the different data sets/machines) which contradict the results that we can see from Leto; most importantly being that in all cases (e.g. with/without threads) 6.6.0 proves to export data faster than 6.5.0 does.

          The difference in throughput for cbexport may simply be down to the performance issues that we've been seeing on the Leto. There are some daily tests which are run on triton which includes a backup. We can see the throughput for this backup is significantly higher than on Leto. Further investigation shows that the throughput hasn't degraded on the triton cluster since the daily performance tests began for 7.0.0. It might be interesting if we could run a test comparing 6.5.0/6.6.0 cbexport on an alternative cluster so that we can determine if there is actually an issue in cbexport.

          Thanks in advance,
          James

          Show
          james.lee James Lee added a comment - Hi Bo-Chun Wang , I've been having a little look into why this might be happening because there haven't been many (if any) major code changes to cbexport between those two builds; the only thing I can think of is that by default we will now request that Couchbase Server sequentially backfills the vBuckets. I have my doubts whether this may be the cause of the issue because if we look at the latest backup to blackhole test we can see that the throughput has increased (backup/export share the same code in this respect). I did some local testing of cbexport (which should be taken with a grain of salt due to the different data sets/machines) which contradict the results that we can see from Leto; most importantly being that in all cases (e.g. with/without threads) 6.6.0 proves to export data faster than 6.5.0 does. The difference in throughput for cbexport may simply be down to the performance issues that we've been seeing on the Leto. There are some daily tests which are run on triton which includes a backup. We can see the throughput for this backup is significantly higher than on Leto. Further investigation shows that the throughput hasn't degraded on the triton cluster since the daily performance tests began for 7.0.0. It might be interesting if we could run a test comparing 6.5.0/6.6.0 cbexport on an alternative cluster so that we can determine if there is actually an issue in cbexport. Thanks in advance, James
          Hide
          bo-chun.wang Bo-Chun Wang added a comment -

          We re-ran these tests and found the performance is back or even better than baseline. The regression we saw may be a false regression.

           

          Export JSON Lines:

          Build

          Avg. MB/sec

          Job  
          7.0.0-2278 128 http://perf.jenkins.couchbase.com/job/leto/13309/ 1st run
          7.0.0-2278 158 http://perf.jenkins.couchbase.com/job/leto/13358/ 2nd run

           

          Build

          Avg. MB/sec

          Job  
          6.6.0-7737 115 http://perf.jenkins.couchbase.com/job/leto/13202/ 1st run
          6.6.0-7737 159 http://perf.jenkins.couchbase.com/job/leto/13366/ 2nd run

          Export JSON List: 

          Build

          Avg. MB/sec

          Job  
          7.0.0-2278 131 http://perf.jenkins.couchbase.com/job/leto/13311/ 1st run
          7.0.0-2278 150 http://perf.jenkins.couchbase.com/job/leto/13359/ 2nd run

           

          Build

          Avg. MB/sec

          Job  
          6.6.0-7737 100 http://perf.jenkins.couchbase.com/job/leto/13203/ 1st run
          6.6.0-7737 156 http://perf.jenkins.couchbase.com/job/leto/13354/ 2nd run
          Show
          bo-chun.wang Bo-Chun Wang added a comment - We re-ran these tests and found the performance is back or even better than baseline. The regression we saw may be a false regression.   Export JSON Lines: Build Avg. MB/sec Job   7.0.0-2278 128 http://perf.jenkins.couchbase.com/job/leto/13309/ 1st run 7.0.0-2278 158 http://perf.jenkins.couchbase.com/job/leto/13358/ 2nd run   Build Avg. MB/sec Job   6.6.0-7737 115 http://perf.jenkins.couchbase.com/job/leto/13202/ 1st run 6.6.0-7737 159 http://perf.jenkins.couchbase.com/job/leto/13366/ 2nd run Export JSON List:   Build Avg. MB/sec Job   7.0.0-2278 131 http://perf.jenkins.couchbase.com/job/leto/13311/ 1st run 7.0.0-2278 150 http://perf.jenkins.couchbase.com/job/leto/13359/ 2nd run   Build Avg. MB/sec Job   6.6.0-7737 100 http://perf.jenkins.couchbase.com/job/leto/13203/ 1st run 6.6.0-7737 156 http://perf.jenkins.couchbase.com/job/leto/13354/ 2nd run

            People

            Assignee:
            bo-chun.wang Bo-Chun Wang
            Reporter:
            bo-chun.wang Bo-Chun Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty