Add support for /sys/devices/system/node and subfiles

[Hunter1016]

No description provided.

[brauner]

In general, I think this is fine but should we maybe introduce a new command line switch so users can choose whether or not they want this emulation? //Cc @stgraber

[Hunter1016]

In general, I think this is fine but should we maybe introduce a new command line switch so users can choose whether or not they want this emulation? //Cc @stgraber

For whether or not want this emulation, like /sys/devices/system/cpu emulation, users can choose not to use by just not mounting it into the container.
Do you mean a parameter like '--enable-cfs'? Different from cpu, which can be controlled by both cpuset.cpus and cpu quota/period, node only lies on cpuset, cpuset.cpus and cpuset.mems, therefore, I can't think of a similar parameter.
THX.

[stgraber]

@brauner I can't think of a good reason to disable this one. If someone comes up with a good reason to turn it off, I think I'd prefer we switch to a --disable-emulation=KEYWORD,... type list so one can disable each file/section independently without us having to add a ton of new flags.

[brauner]

@brauner I can't think of a good reason to disable this one. If someone comes up with a good reason to turn it off, I think I'd prefer we switch to a --disable-emulation=KEYWORD,... type list so one can disable each file/section independently without us having to add a ton of new flags.

I'm just a bit concerned that we're too lax with new features here and might introduce regressions. That's the main driver for me requesting this.

[stgraber]

I'm just a bit concerned that we're too lax with new features here and might introduce regressions. That's the main driver for me requesting this.

Well, this isn't going to be in stable-5.0 as it's a new feature, so I'm not too worried about that.

[Hunter1016]

@brauner @stgraber Based on above discussions, I think disabling this emulation can be easily and flexibly done by just not mount it into the container. Each container has the right to choose.
Conversely, if we introduce a new flag, it becomes a global option that all containers should follow.

[mihalicyn]

Hi @Hunter1016

Really sorry for a long delay with reviewing this. Could you rebase it?

We have just discussed with Stéphane and we would like to merge it and have it in an upcoming LXCFS release.

Kind regards,
Alex

[stgraber]

We'll go with a --enable-cpu-nodes flag for the time being and then replace that with the per-container config stuff a bit later (after 6.0 LTS).

[mihalicyn]

JFYI: Right now I'm working on rebasing this on top of main branch.

[mihalicyn]

replaced by #632

[mihalicyn]

This thing is tightly coupled with /sys/devices/system/cpu virtualization which was removed recently in scope of #557

If I fully understand the idea of this patchset is to hide NUMA nodes in accordance with what we have in cpuset in cgroup.

Taking into account the problem described by @tych0 in #557 I'm not sure that it's a good idea in general, because user space application may not expect this.

It can lead to weird configurations like this (manually crafted listing example, it's not from a real system):

$ ls -la /sys/devices/system/node/
total 0
drwxr-xr-x  4 root root    0 Mar 27 08:39 .
drwxr-xr-x 10 root root    0 Mar 27 08:39 ..
<...>
drwxr-xr-x  6 root root    0 Mar 27 08:39 node0
drwxr-xr-x  6 root root    0 Mar 27 08:39 node2
-r--r--r--  1 root root 4096 Mar 27 12:47 online
<..>

Dear @Hunter1016, couldn't you provide a bit more context why this was implemented? Do you have any use-case example for this?

[tych0]

I agree; we should only emulate what the kernel does. Anything beyond that will lead to problems.

[Hunter1016]

This thing is tightly coupled with /sys/devices/system/cpu virtualization which was removed recently in scope of #557

If I fully understand the idea of this patchset is to hide NUMA nodes in accordance with what we have in cpuset in cgroup.

Taking into account the problem described by @tych0 in #557 I'm not sure that it's a good idea in general, because user space application may not expect this.

It can lead to weird configurations like this (manually crafted listing example, it's not from a real system):

$ ls -la /sys/devices/system/node/
total 0
drwxr-xr-x  4 root root    0 Mar 27 08:39 .
drwxr-xr-x 10 root root    0 Mar 27 08:39 ..
<...>
drwxr-xr-x  6 root root    0 Mar 27 08:39 node0
drwxr-xr-x  6 root root    0 Mar 27 08:39 node2
-r--r--r--  1 root root 4096 Mar 27 12:47 online
<..>

Dear @Hunter1016, couldn't you provide a bit more context why this was implemented? Do you have any use-case example for this?

@mihalicyn I'm sorry for a bit late, the most important motivation is running applications depending on libhwloc meets segmentaion fault because it reads /sys/devices/system/node/node%u/cpumap and /sys/devices/system/cpu/online to analyze hardware topology.

[mihalicyn]

I'm sorry for a bit late, the most important motivation is running applications depending on libhwloc meets segmentaion fault because it reads /sys/devices/system/node/node%u/cpumap and /sys/devices/system/cpu/online to analyze hardware topology.

Hi @Hunter1016,

sorry for my very late response. That makes sense, so if I understand you correctly, anything that depends on libhwloc will crash these days inside a container which uses LXCFS? That's bad. At the same time, we have learned a lesson in a hard way, that messing with cpu/numa report information can be quite dangerous, confuse user space application. So before changing anything here, we should clearly describe what specifically we are trying to emulate and how, cause otherwise we might end up having a broken emulation for a kernel ABI. I also have done some work on #638 which is about making a way to dynamically enable/disable some LXCFS features on per-container basis. We could consider adding this stuff as a opt-in feature.

Please, let me know if you are still interested in this stuff.

Kind regards,
Alex