Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.0.0
-
Triaged
-
Yes
Description
We need to calculate the right number of cores configured by the container environment and ensure all services use it. Since NS-SERVER is the bootstrapping all services, we'll need to add logic to detect and pass on the proper args/env as appropriate to the type of runtime a service is using.
Most of our services uses a some flavor of managed virtual machine (Java, Golang, Erlang). These are the actual components that automatically detects number of cores, based on whichever calls they make to the OS. Some may offer a way to manually force number of cores avail to the virtual machine. Each service will need to provide bootstrapping guidance how to limit theirs. Then, we can add a wrapper to the bootstrapping that detects whether or not, cgroups is in use and calculate the cores made available to the container and pass it on to the startup script of each service and use the proper arguments to restrict the number of cores. Since most services are started by the babysitter, this can be done centrally by ns_server.
Java : command line -XX:ActiveProcessorCount=<cores>
Golang : environment variable GOMAXPROCS=<cores>
Erlang: command line +S <cores>:<cores>
C/C++: No Virtual Machine. cgroup detection logic needs to be prescribed in code.
Important to note, as the number of threads created by each service may, or may not be derived by the number of available cores, each service will need to adjust their logic of thread allocation. In the case of Erlang based processes (NS_SERVER, COUCHDB), there will be one thread per core by default. So adjusting number of cores will also impact number of threads.
Attachments
Issue Links
- blocks
-
CMOS-233 Memory statistics in containers use those of the host
- To Do
-
K8S-2724 Add tests for cgroup/COUCHBASE_CPU_COUNT changes
- In Progress
- depends on
-
MB-52029 Eventing Service should honor the CPU & Memory limits set in cgroups
- Resolved
-
MB-40464 CPU statistics in containers limited via CFS are misleading
- Closed
-
MB-49632 cb::get_available_cpu_count ignores CPU quota
- Closed
-
MB-51864 The Backup Service should honor the GOMAXPROCS env var if specified
- Closed
-
MB-51866 [cbas] Respect COUCHBASE_CPU_COUNT env (set max num of threads for JVM)
- Closed
- relates to
-
MB-52686 cbbackupmgr does not correctly recognise container limits
- Closed
-
MB-48247 Account for container memory and cpu limits in plasma storage
- Closed
-
MB-49381 Projector memory monitoring doesn't work correctly in memory limited containers
- Closed
-
MB-40464 CPU statistics in containers limited via CFS are misleading
- Closed
-
MB-48873 Consider runtime.GOMAXPROCS(0) vs runtime.NumCPU()
- Closed
-
MB-48247 Account for container memory and cpu limits in plasma storage
- Closed
-
MB-49382 [CBSE][sigar] Indexer CPU and memory monitoring doesn't work correctly in CPU limited containers
- Closed
-
K8S-2514 [Self Cert Tool] Enhance PreFlight checks to get actual count of PID limit on containers.
- Open
- links to
- mentioned in
-
Page Loading...
For Gerrit Dashboard: MB-47591 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
167710,11 | MB-47591: Support for sigar v6 | master | ns_server | Status: MERGED | +2 | +1 |
167712,15 | MB-47591: [babysitter] Set COUCHBASE_CPU_COUNT when starting | master | ns_server | Status: MERGED | +2 | +1 |
167713,15 | MB-47591: Set the number of schedulers for ... | master | ns_server | Status: MERGED | +2 | +1 |
167809,11 | MB-47591: Don't pass COUCHBASE_CPU_COUNT to sigar... | master | ns_server | Status: ABANDONED | 0 | 0 |
174433,3 | MB-47591: [BP] Move basic sigar functions to sigar.erl ... | neo | ns_server | Status: MERGED | +2 | +1 |
174434,3 | MB-47591: [BP] [babysitter] Set COUCHBASE_CPU_COUNT when starting | neo | ns_server | Status: MERGED | +2 | +1 |
174435,3 | MB-47591: [BP] Set the number of schedulers for ... | neo | ns_server | Status: MERGED | +2 | +1 |
174436,3 | MB-47591: [sigar] Use uint64 for all cgroups fields | neo | ns_server | Status: ABANDONED | +2 | 0 |
174437,2 | MB-47591: Use uint64 for all cgroups fields | neo | sigar | Status: ABANDONED | +2 | 0 |
174438,5 | MB-47591: Use cgroups data from sigar for stats | neo | ns_server | Status: MERGED | +2 | +1 |
174439,5 | MB-47591: Move start of sigar to a separate process ... | neo | ns_server | Status: MERGED | +2 | +1 |
174440,5 | MB-47591: Introduce sigar:get_gauges/1 | neo | ns_server | Status: MERGED | +2 | +1 |
174441,5 | MB-47591: Don't read cpu and memory cgroups limit from files | neo | ns_server | Status: MERGED | +2 | +1 |
174442,5 | MB-47591: Add some sys metrics that honor cgroups limits | neo | ns_server | Status: MERGED | +2 | +1 |
174479,3 | MB-47591: [sigar] Take care of paddings when... | neo | ns_server | Status: MERGED | +2 | +1 |
174553,2 | Merge remote-tracking branch 'couchbase/neo' | master | ns_server | Status: MERGED | +2 | +1 |
174559,3 | MB-47591: Fix ns_config_default:all_upgrades_test_ unit test | neo | ns_server | Status: MERGED | +2 | +1 |
174599,1 | Merge branch 'neo' | master | ns_server | Status: MERGED | +2 | +1 |
174609,1 | MB-47591: Add cpu_host_cores_available metric | master | ns_server | Status: ABANDONED | 0 | 0 |
174610,2 | MB-47591: Add cpu_host_cores_available metric | neo | ns_server | Status: MERGED | +2 | +1 |
174613,1 | Merge remote-tracking branch 'couchbase/neo' | master | ns_server | Status: MERGED | +2 | +1 |