Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-52686

cbbackupmgr does not correctly recognise container limits

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.2
    • 7.1.1
    • tools
    • Untriaged
    • 1
    • No

    Description

      Hi,

      I'm testing container limits in a Docker container, started with the command:
      docker run -d -m 1g --cpus="1" --name db -p 8091-8094:8091-8094 -p 11210:11210 registry.gitlab.com/cb-vanilla/server:7.1.1-3135
      The docker docs suggests that this should limit the container to 1 CPU and 1GB memory. I can check the /metrics endpoint and see that ns_server is seeing the container limit:

      sys_cpu_cores_available{category="system"} 1.00000000000000000000e+00
      

      I then (via the UI @ localhost:8091) load travel-sample. Then, manually invoking cbbackupmgr, I do the following to backup:
      cbbackupmgr config --archive /data/backups --repo example
      cbbackupmgr backup -a /data/backups -r example -c couchbase://localhost -u Administrator -p password
      The backup exits partway through, reporting Killed:

      Backing up to '2022-06-23T16_23_04.0768512Z'
      Transferring key value data for bucket 'travel-sample' at 984.39KiB/s (about 20s remaining)                            40541 items / 26.56MiB
      [===============================================================================                                                     ] 59.96%
      Killed
      

      docker stats shows it using everything it's assigned, so I assume it is resource killed.

      CONTAINER ID   NAME                 CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
      a4cdd969fdf1   db                   108.66%   1023MiB / 1GiB       99.94%    3.99MB / 8.42MB   4.36GB / 9.22GB   305
      

      Running dmesg in the Docker VM shows:

      [113262.708414] Memory cgroup out of memory: Killed process 223576 (cbbackupmgr) total-vm:3646456kB, anon-rss:836468kB, file-rss:28344kB, shmem-rss:0kB, UID:0 pgtables:4648kB oom_score_adj:0
      [113262.855543] oom_reaper: reaped process 223576 (cbbackupmgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB 

       

      If I then run cbbackupmgr collect-logs -a /data/backups/, the first line of the logs shows it is expecting the full resources of my Docker Desktop VM:

      2022-06-23T15:30:57.144+00:00 (Cmd) cbbackupmgr version 7.1.1-3135 Hostname: a4cdd969fdf1 OS: linux Version: 5.10.76-linuxkit Arch: amd64 vCPU: 4 Memory: 12561879040 (11.70GiB)
      

      Manually setting GOMAXPROCS=1 results in the logs recording the correct vCPU count, but still being killed before the backup completes. I have attached the logs for the GOMAXPROCS=1 run to this issue.

      Further info:
      This is happening in the 7.1.1-3135 container built by the build team, available on GitLab (where all pre-release containers go). As far as I can tell, this container runs Ubuntu 20.04:

      cat /etc/issue
      Ubuntu 20.04.4 LTS \n \l
      

      ...and has cgroup2 enabled:

      grep cgroup /proc/filesystems
      nodev	cgroup
      nodev	cgroup2
      

      Attachments

        1. backup-0.log
          291 kB
        2. cbbackup-windows.log
          371 kB
        3. windows-output.txt
          78 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              gilad.kalchheim Gilad Kalchheim
              roo.thorp Roo Thorp
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty