Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.1
-
Untriaged
-
1
-
No
Description
Hi,
I'm testing container limits in a Docker container, started with the command:
docker run -d -m 1g --cpus="1" --name db -p 8091-8094:8091-8094 -p 11210:11210 registry.gitlab.com/cb-vanilla/server:7.1.1-3135
The docker docs suggests that this should limit the container to 1 CPU and 1GB memory. I can check the /metrics endpoint and see that ns_server is seeing the container limit:
sys_cpu_cores_available{category="system"} 1.00000000000000000000e+00 |
I then (via the UI @ localhost:8091) load travel-sample. Then, manually invoking cbbackupmgr, I do the following to backup:
cbbackupmgr config --archive /data/backups --repo example
cbbackupmgr backup -a /data/backups -r example -c couchbase://localhost -u Administrator -p password
The backup exits partway through, reporting Killed:
Backing up to '2022-06-23T16_23_04.0768512Z' |
Transferring key value data for bucket 'travel-sample' at 984.39KiB/s (about 20s remaining) 40541 items / 26.56MiB |
[=============================================================================== ] 59.96% |
Killed
|
docker stats shows it using everything it's assigned, so I assume it is resource killed.
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
|
a4cdd969fdf1 db 108.66% 1023MiB / 1GiB 99.94% 3.99MB / 8.42MB 4.36GB / 9.22GB 305 |
Running dmesg in the Docker VM shows:
[113262.708414] Memory cgroup out of memory: Killed process 223576 (cbbackupmgr) total-vm:3646456kB, anon-rss:836468kB, file-rss:28344kB, shmem-rss:0kB, UID:0 pgtables:4648kB oom_score_adj:0 |
[113262.855543] oom_reaper: reaped process 223576 (cbbackupmgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB |
If I then run cbbackupmgr collect-logs -a /data/backups/, the first line of the logs shows it is expecting the full resources of my Docker Desktop VM:
2022-06-23T15:30:57.144+00:00 (Cmd) cbbackupmgr version 7.1.1-3135 Hostname: a4cdd969fdf1 OS: linux Version: 5.10.76-linuxkit Arch: amd64 vCPU: 4 Memory: 12561879040 (11.70GiB) |
Manually setting GOMAXPROCS=1 results in the logs recording the correct vCPU count, but still being killed before the backup completes. I have attached the logs for the GOMAXPROCS=1 run to this issue.
Further info:
This is happening in the 7.1.1-3135 container built by the build team, available on GitLab (where all pre-release containers go). As far as I can tell, this container runs Ubuntu 20.04:
cat /etc/issue
|
Ubuntu 20.04.4 LTS \n \l |
...and has cgroup2 enabled:
grep cgroup /proc/filesystems
|
nodev cgroup
|
nodev cgroup2
|