Wrong cpu number reported, when only CFS (cpu.max) cgroup2 limit is set

[okhowang]

cpu count was read in current cgroup only.
if parent cgroup limit cpu to 2, but current cgroup limit is unlimit,
lxcfs will show all cpu for container.

[mihalicyn]

Hi @okhowang,

cpu count was read in current cgroup only.

I'm not sure what "current" means in this context, because if we speak about "current" like in the kernel, then it is not true. Actually, we look for initial process in the container and then take its cgroup and take all the limits from it.

if parent cgroup limit cpu to 2, but current cgroup limit is unlimit,
lxcfs will show all cpu for container.

I can imagine that in some configurations it might be a problem, but instead of me guessing, please can you show the entire reproducer for your setup and describe in detail what you are getting and what you are expecting to get from LXCFS in your case.

Kind regards,
Alex

[okhowang]

assume lxcfs run on /var/lib/lxcfs on hosts
lxcfs -l --enable-cfs /var/lib/lxcfs

start a Dind with cpu=2
docker run -it --privileged --cpus 2 --rm --name dind -v /var/lib/lxcfs:/var/lib/lxcfs:rshared docker:dind

dind container will create a init cgroup
like 0::/system.slice/docker-88f6856c77a749f88a92fdb103939895486fb47dc71c70910955b06409504994.scope/init
but docker run --cpus 2 set cpu.max on /system.slice/docker-88f6856c77a749f88a92fdb103939895486fb47dc71c70910955b06409504994.scope

$ cat /sys/fs/cgroup/system.slice/docker-88f6856c77a749f88a92fdb103939895486fb47dc71c70910955b06409504994.scope/cpu.max
200000 100000
$ cat /sys/fs/cgroup/system.slice/docker-88f6856c77a749f88a92fdb103939895486fb47dc71c70910955b06409504994.scope/init/cpu.max
max 100000

run in init cgroup (32 cpu on host)

$ docker exec -it dind cat /var/lib/lxcfs/proc/stat
cpu  479296775 64555 225629081 8578667942 51804283 0 19487016 0 0 0
cpu0 16627746 1957 7118535 262164690 1650779 0 9366513 0 0 0
cpu1 14798361 1517 6775671 267360763 1284331 0 4204927 0 0 0
cpu2 17628299 2170 7784110 263619622 1720743 0 1012459 0 0 0
cpu3 13749285 970 5939891 270667403 1315381 0 881009 0 0 0
cpu4 17561583 2781 7676893 264507718 1729131 0 371456 0 0 0
cpu5 13823233 795 5978830 271023330 1331979 0 303458 0 0 0
cpu6 17561536 2549 7667983 264598291 1741211 0 266904 0 0 0
cpu7 13898293 889 5988255 270884185 1326317 0 223024 0 0 0
cpu8 16732585 3402 8183687 264637296 2043271 0 269231 0 0 0
cpu9 13174499 599 6292493 271098081 1453613 0 215797 0 0 0
cpu10 16330083 3491 8076792 265406116 1819326 0 166965 0 0 0
cpu11 13014857 1408 6135051 271979356 1282101 0 106027 0 0 0
cpu12 16255329 2601 7999505 265617977 1678395 0 148293 0 0 0
cpu13 13117095 568 6266076 271600290 1205003 0 104315 0 0 0
cpu14 16169995 2378 7981995 266049512 1481584 0 140764 0 0 0
cpu15 13616012 1347 6320179 269200810 2809304 0 93250 0 0 0
cpu16 16342396 3634 8209222 265341159 1776820 0 150896 0 0 0
cpu17 13229924 1114 6344031 271180005 1331301 0 97178 0 0 0
cpu18 16265812 2709 8032657 265247634 1826036 0 151311 0 0 0
cpu19 13022566 763 6094823 272080211 1325953 0 75511 0 0 0
cpu20 16208191 3134 7954160 265873239 1797022 0 136376 0 0 0
cpu21 13148876 1774 6153174 271640349 1342517 0 80249 0 0 0
cpu22 16198914 3380 8060963 265860584 1809741 0 132527 0 0 0
cpu23 13162773 542 6148830 271897792 1322667 0 70911 0 0 0
cpu24 16533972 3041 8072035 265111828 2073439 0 118196 0 0 0
cpu25 13072590 1325 6167916 271837801 1369976 0 64961 0 0 0
cpu26 16150019 2516 7939378 265842372 1851249 0 117020 0 0 0
cpu27 13104319 1080 6136248 271900614 1284770 0 57582 0 0 0
cpu28 16208665 3121 7932671 265886367 1725558 0 116728 0 0 0
cpu29 13088320 646 6113671 271943541 1226479 0 55337 0 0 0
cpu30 16366834 3284 7986508 265116446 2170179 0 116538 0 0 0
cpu31 13133813 3070 6096848 271492560 1698107 0 71303 0 0 0
intr 99751113243 172 9 0 0 396 0 3 0 0 0 0 0 0 0 2877461 0 0 0 0 0 0 0 0 0 0 155572439 0 126253132 121359076 0 28286111 29904748 0 23005366 2 9102690 2 36384054 2 11420618 2 36169003 2 32773259 3 6683614 3 6648852 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 154831916344
btime 1767167954
processes 333942856
procs_running 3
procs_blocked 1
softirq 43369382359 1 970839002 666550 1918802161 301608625 0 745422 3218372083 727 2598609420 0

[mihalicyn]

Hi @okhowang,

now it starts to makes sense, thank you for providing a reproducer and describing your use-case.

I've changed the title of the issue to "Wrong cpu number reported, when only CFS (cpu.max) cgroup2 limit is set", cause it better describes what we have here.

1. In your case, you use cgroup2 and set cpu.max parameter, but not cpuset.cpus. Our current logic seems not correct when we deal with cpu.max, while it looks correct for cpuset.cpus. Just compare cgfsng_get_cpuset_cpus and read_cpu_cfs_param.

2. You can use a workaround (avoid CFS parameters and use cpuset), by explicitly specifying cpus to docker like --cpuset-cpus 2,3,5 instead of --cpus.

[mihalicyn]

I have assigned you to this issue, because you've done a lot of work already. Let me know if you don't have a time to push this forward.