Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-11405

Shared thread pool: high CPU overhead due to OS level context switches / scheduling

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0
    • 3.0
    • couchbase-bucket
    • Security Level: Public
    • Build 3.0.0-805

      Platform = Physical
      OS = CentOS 6.5
      CPU = Intel Xeon E5-2680 v2 (40 vCPU)
      Memory = 256 GB
      Disk = RAID 10 SSD

    Description

      Originally reported as "~2400% CPU consumption by memcached during ongoing workload with five (5) buckets ".

      The CPU usage for memcached process is more than two times the usage in the previous release. This is due to increased scheduling overhead from the shared thread pool.
      Workaround: Reduce the number of threads on systems that have more than 30 cores.

      2 nodes, 5 buckets
      1M docs (clusterwise), equally distributed, non-DGM
      10K mixed ops/sec (85% reads, 1% creates, 1% deletes, 13% updates; clusterwise), equally distributed

      CPU utilization in 2.5.1: ~300%
      CPU utilization in 3.0.0: ~2400%

      Attachments

        1. cpu.png
          cpu.png
          12 kB
        2. max_threads_cpu.png
          max_threads_cpu.png
          45 kB
        3. memcached_cpu_b988.png
          memcached_cpu_b988.png
          46 kB
        4. memcached_cpu_toy.png
          memcached_cpu_toy.png
          62 kB
        5. perf_b829.log
          3.60 MB
        6. perf_b854_8threads.log
          3.48 MB
        7. perf.log
          3.92 MB

        Issue Links

          For Gerrit Dashboard: MB-11405
          # Subject Branch Project Status CR V

          Activity

            People

              pavelpaulau Pavel Paulau (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty