runc use systemd driver cause oom event lost because file memory.events delete too quickly

[ningmingxiao]

Description

no such file or directory path is /sys/fs/cgroup/test.slice/cri-containerd-809b5245695a19958297daee00ccbb7dbb7a21e4457757f209b931e81b88fd88.scope/memory.events

Steps to reproduce the issue

1.run critest many time with containerd
the ci meet this bug containerd/containerd#12681

Describe the results you received and expected

memory.events delete can add some sleep

What version of runc are you using?

any

Host OS information

No response

Host kernel information

No response

[ningmingxiao]

If runc use systemd driver and container init process is oom killed, will systemd delete cgroup even shim is reading oom events from file memory.events? It may cause oom event lost .@kolyshkin @cyphar

[kolyshkin]

1. No one removes the memory.events file, it's part of cgroup. Once cgroup is removed, this file is removed, too.
2. Unless runc delete is called, runc does not remove container's cgroup.
3. Yes, I'm afraid this is systemd removing the cgroup once the last task (or the init task) has exited. You may try setting RemainAfterExit=yes property to change that.

If you still believe this is runc bug, can you please provide a reproducer that only uses runc (i.e. no higher level runtimes), @ningmingxiao?

[ningmingxiao]

should I use RemainAfterExit=yes in containerd service? RemainAfterExit=yes may cause cgroup leak. @kolyshkin
I add systemdDbus.PropRemainAfterExit https://github.com/opencontainers/runc/blob/v1.3.4/vendor/github.com/opencontainers/cgroups/systemd/v2.go#L298

	properties = append(properties, systemdDbus.PropDescription("libcontainer container "+c.Name))
	properties = append(properties, systemdDbus.PropRemainAfterExit(true))

but failed

unable to apply cgroup configuration: unable to start unit "cri-containerd-1698e7828d77d13e7212b1685e75eae85e27be3f44c2169a8ae1993d05ad2215.scope" (properties [{Name:Description Value:"libcontainer container 1698e7828d77d13e7212b1685e75eae85e27be3f44c2169a8ae1993d05ad2215"} {Name:RemainAfterExit Value:true} {Name:Slice Value:"test.slice"} {Name:Delegate Value:true} {Name:PIDs Value:@au [120903]} {Name:MemoryAccounting Value:true} {Name:CPUAccounting Value:true} {Name:IOAccounting Value:true} {Name:TasksAccounting Value:true} {Name:DefaultDependencies Value:false}]): Cannot set property RemainAfterExit, or unknown property."

[ningmingxiao]

maybe because scope (use *.slice ) doesn't support RemainAfterExit=yes.

[ningmingxiao]

runc events may lost oom event @kolyshkin
I create a it in containerd containerd/containerd#12696 if add containers num to 100 it's easy to reproduce. or we should create a pr in systemd to add a sleep property before delete cgroup, but I'm not familar with systemd code.