Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45547

covering array index latency increase after 2 query commits

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Looks like there are two commits which are causing higher latency across all percentiles for this test: 90th percentile query latency (ms), Q2 Array Indexing, Covering, Singleton Unique Lookup, 450 queries/sec

      http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/array/all

      4735 - http://perf.jenkins.couchbase.com/job/iris/24899/ - 6.3
      4745 - http://perf.jenkins.couchbase.com/job/iris/25031/ - 10.6
      4750 - http://perf.jenkins.couchbase.com/job/iris/25033/ - 6.9
      4753 - http://perf.jenkins.couchbase.com/job/iris/25034/ - 8.0 <--
      4754 - http://perf.jenkins.couchbase.com/job/iris/25036/ - 6.8 <--
      4755 - http://perf.jenkins.couchbase.com/job/iris/25032/ - 311 <--
      4765 - http://perf.jenkins.couchbase.com/job/iris/25025/ - 322
      4797 - http://perf.jenkins.couchbase.com/job/iris/24992/ - 309

      There are two commits, one in 4574 and the other in 4575. The first commit causes queries to begin to timeout (4 minutes) but no overall increase in latency. The second commit further causes increase in latency.

      http://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=7.0.0&fromBuild=4753&toVersion=7.0.0&toBuild=4755&f_query=on&f_query-ee=on

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_700-4753_access_07ed&snapshot=iris_700-4755_access_2799

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            jliang John Liang added a comment -

            File MB-45873 for the disk space issue.

            Regarding high CPU, I think I find out what it is by looking at the logs. Korrigan Clark you don't have to run cpu profile.

            jliang John Liang added a comment - File MB-45873 for the disk space issue. Regarding high CPU, I think I find out what it is by looking at the logs. Korrigan Clark you don't have to run cpu profile.
            jliang John Liang added a comment -

            This is the latest run (1 hour run). 90 percentile latency is now 9ms. Besides that, latency distribution is more gradual than 6.6.x release (99 percentile latency is around 14ms). Memory usage is below quota (in 6.6.x, memory usage is above quota).

            The downside is higher CPU, but I think it is due to different issue. Looking at CPU profile, over 50% of CPU is spent on mcall. This test uses a large index and it can consumes a lot of IO bandwidth. There is not enough IO bandwidth to compact the log on disk. As the log size grows, the writer is going to get throttled by yielding CPU often. This leads to higher CPU on mcall. I am going to open another MB to track this issue.

            Overall, we perform better in 7.0 but we have to figure out what happen to mcall for large key.

            http://perf.jenkins.couchbase.com/job/aether/191/

            jliang John Liang added a comment - This is the latest run (1 hour run). 90 percentile latency is now 9ms. Besides that, latency distribution is more gradual than 6.6.x release (99 percentile latency is around 14ms). Memory usage is below quota (in 6.6.x, memory usage is above quota). The downside is higher CPU, but I think it is due to different issue. Looking at CPU profile, over 50% of CPU is spent on mcall. This test uses a large index and it can consumes a lot of IO bandwidth. There is not enough IO bandwidth to compact the log on disk. As the log size grows, the writer is going to get throttled by yielding CPU often. This leads to higher CPU on mcall. I am going to open another MB to track this issue. Overall, we perform better in 7.0 but we have to figure out what happen to mcall for large key. http://perf.jenkins.couchbase.com/job/aether/191/
            jliang John Liang added a comment -

            push to master branch

            jliang John Liang added a comment - push to master branch

            Build couchbase-server-7.0.0-5070 contains plasma commit 499d4e1 with commit message:
            MB-45547: Use memory stats for throttling scan traffic

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5070 contains plasma commit 499d4e1 with commit message: MB-45547 : Use memory stats for throttling scan traffic
            korrigan.clark Korrigan Clark (Inactive) added a comment - verifying:  http://perf.jenkins.couchbase.com/job/iris/25194/

            People

              jliang John Liang
              korrigan.clark Korrigan Clark (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty