Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49501

Indexer cpu throttle cuts query throughput by 47%

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Unknown

    Description

      There was a previous issue opened https://issues.couchbase.com/browse/MB-49301 but resolved as not a bug. Maybe more details of the CPU throttle algorithm could help understand the result better and how its intended to work. When I set the throttle level to 0.95, query throughput gets reduced by 40%. The performance test pushes indexer cpu normally close to 100%. If the throttle level is set to 0.95 why does cpu usage get throttled down to 60% instead of right below 95%. It seems there is a lot of wasted cpu cycles here.

      Comparing these two test on 7.1.0-1650:

      http://showfast.sc.couchbase.com/#/timeline/Linux/n1ql/Q5_Q7/all

      Avg. Query Throughput (queries/sec), CI6, Group By Query (1K matches), MOI, not_bounded, s=1 c=1 i=1

      http://perf.jenkins.couchbase.com/job/iris-multi-client/12973/ - 15280

      Avg. Query Throughput (queries/sec), CI6, Group By Query (1K matches), MOI, not_bounded, Indexer CPU Throttle 0.95, s=1 c=1 i=1

      http://perf.jenkins.couchbase.com/job/iris-multi-client/13027/  - 7651

      graph comparison: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_710-1650_access_0436&snapshot=iris_710-1650_access_afdf

      Indexer cpu drops from 4800% to 3000%

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_710-1650_access_0436&snapshot=iris_710-1650_access_afdf#0103718d11a8aa31d5c328c63302ff1d

       

      Logs from throttle run (.45 is the index node):

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.45.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.55.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.70.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.71.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.72.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.73.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-iris-multi-client-13027/172.23.100.9.zip

       

      Is this really the intended behaviour? This seems like it could have negative side effects when upgrading a cluster if this is the default setting.

       

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-49501
          # Subject Branch Project Status CR V

          Activity

            Build couchbase-server-7.1.0-2049 contains indexing commit c9aeea0 with commit message:
            MB-49501 Part 6 (7.1.0 2037): Shorten HealthCheck "Slow call" threshold

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2049 contains indexing commit c9aeea0 with commit message: MB-49501 Part 6 (7.1.0 2037): Shorten HealthCheck "Slow call" threshold
            korrigan.clark Korrigan Clark (Inactive) added a comment - http://perf.jenkins.couchbase.com/job/iris-multi-client/14318/

            Resolve this now because:

            1. Throttling parameters changed so throttling is much lighter.
            2. Throttling is now disabled by default.
            3. If we reenable it, it will be at CPU target 0.98 (98%), which with current parameters had only a small (~0.5%) performance impact on the test for which this MB was opened.
            kevin.cherkauer Kevin Cherkauer (Inactive) added a comment - Resolve this now because: Throttling parameters changed so throttling is much lighter. Throttling is now disabled by default. If we reenable it, it will be at CPU target 0.98 (98%), which with current parameters had only a small (~0.5%) performance impact on the test for which this MB was opened.

            Build couchbase-server-7.1.0-1883 contains indexing commit 9cc3bff with commit message:
            MB-49501 Part 5 (7.1.0 1861): Log long heartbeats; reduce dump frequency

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1883 contains indexing commit 9cc3bff with commit message: MB-49501 Part 5 (7.1.0 1861): Log long heartbeats; reduce dump frequency

            Build couchbase-server-7.1.0-1867 contains indexing commit 58027c0 with commit message:
            MB-49501 Part 4 (7.1.0 1861): Temp disable CPU throttling for perf tests

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1867 contains indexing commit 58027c0 with commit message: MB-49501 Part 4 (7.1.0 1861): Temp disable CPU throttling for perf tests

            People

              korrigan.clark Korrigan Clark (Inactive)
              korrigan.clark Korrigan Clark (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty