Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40464

CPU statistics in containers limited via CFS are misleading

    XMLWordPrintable

Details

    • Triaged
    • 1
    • Unknown

    Description

      There are a couple of different ways to limit the CPU that can be used by a container.
      A common way used by Docker is specify --cpuset-cpus, which limit the container to only be able to use specific cores within the CPU.
      This is currently auto-detected by ns_server and memcached (MB-28898. MB-20834).

      However, Kubernetes limits CPU for containers using CFS instead (reference: https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu-time-9eff74d3161b).

      Couchbase Server does not account for this, so will incorrectly report the number of CPUs that a Kubernetes pod is using and sigar will also report an incorrect total CPU utilization rate.

      For example, in a container limited to a single CPU core:

      "cpuCount":2
      

      $ cat cpu.cfs_period_us
      100000
      

      $ cat cpu.cfs_quota_us
      100000
      

      I expect that this is not easy to fix, as we'd have to use the CFS quotas and share to calculate the number of CPUs, but raising this for awareness.

      I also suspect that this may also affect the automatic thread generation used by memcached too, but have not verified this.

      Attachments

        Issue Links

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              matt.carabine Matt Carabine (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty