Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.5.1, 6.5.0
-
Inside Docker, macOS as a host, using HPET clock source for Docker VM.
-
1
Description
This is using our official docker image which is built on top of ubuntu 16.04 LTS.
A user has reported this in https://github.com/testcontainers/testcontainers-java/issues/2802 and I've been able to reproduce this locally.
If you create 3-4 buckets, memcached cpu goes up to 100-150% inside the container. This happens with 6.5.0 and 6.5.1. I also tried with 6.0.3 and there the CPU is at 13%.
I triaged a bit with Dave Rigby and the current suspicion is that the sync durability monitor is consuming way more cpu in this setup than tuned for non-virtual linux hosts (less than 1%).. a couple % are probably fine, but this is keeping 1-2 cores busy all the time, even if there is no ops/traffic going through.
If you are not able to reproduce this I can provide the java program to run it, but all testcontainers-java does is spin up the docker container, configure it with alternate addr to expose the ports and then create 3 buckets.
–
Some more info from the initial triage:
top output:
329 couchba+ 20 0 2038928 105812 12564 S 108.7 5.2 7:18.83 memcached
|
points to the NonIO threads:
498 couchba+ 20 0 2038928 104720 12464 S 51.8 5.1 2:14.51 mc:nonIO_1
|
497 couchba+ 20 0 2038928 104720 12464 S 49.2 5.1 2:12.78 mc:nonIO_0
|
mpstat shows we're spending some time in sys:
09:14:34 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
|
09:14:35 all 3.98 0.00 19.38 0.00 0.00 0.00 0.00 0.00 0.00 76.64
|
09:14:35 0 3.16 0.00 11.58 0.00 0.00 0.00 0.00 0.00 0.00 85.26
|
09:14:35 1 0.00 0.00 2.13 0.00 0.00 0.00 0.00 0.00 0.00 97.87
|
09:14:35 2 1.04 0.00 6.25 0.00 0.00 0.00 0.00 0.00 0.00 92.71
|
09:14:35 3 6.25 0.00 40.62 0.00 0.00 0.00 0.00 0.00 0.00 53.12
|
09:14:35 4 7.29 0.00 37.50 0.00 0.00 0.00 0.00 0.00 0.00 55.21
|
09:14:35 5 5.15 0.00 16.49 0.00 0.00 0.00 0.00 0.00 0.00 78.35
|
Attachments
Issue Links
- split to
-
MB-39815 Avoid busy-polling for SyncWrite timeout checking
- Closed
For Gerrit Dashboard: MB-39618 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
129936,2 | MB-39618: Use coarse clock for CappedDurationVBucketVisitor | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
130311,1 | Merge branch mad-hatter into master | master | kv_engine | Status: MERGED | +2 | +1 |
162650,8 | MB-48688: Print clock overheads at startup & via stats | master | kv_engine | Status: MERGED | +2 | +1 |