kubernetes: allow creating privileged pods in the inner cluster - #397
Merged
Conversation
For a privileged pod, runc has to mount a fresh read-write sysfs, which the kernel allows in a mount namespace owned by a user namespace only when the mount namespace already contains a fully-visible read-write sysfs instance. The /sys of the node pods is mounted read-only by the CRI of the outer cluster, and the node pods cannot mount a read-write instance by themselves, so it has to be provided by the host, at /run/usernetes/sysfs, and mounted into the node pods via a hostPath volume. The parent directory is mounted instead of the sysfs itself, as sysfs does not support the idmapped mounts that the kubelet uses for the volumes of UserNS-enabled pods. This issue does not occur in the Docker Compose mode: there the node container runs with `privileged: true` (which is safe under Rootless Docker, as it only grants the capabilities that are namespaced in the RootlessKit user namespace), so Docker mounts its /sys read-write, which already satisfies the visibility requirement. A UserNS-enabled Kubernetes pod cannot be privileged, and the CRI has no option for mounting /sys of an unprivileged pod read-write (unlike `cgroup_writable` for cgroups). Privileged pods with hostNetwork can be created even without this fix, although their /sys is a read-only bind mount of the /sys of the node pod, as mounting a fresh sysfs requires the network namespace to be owned by the user namespace. The network namespace of a node pod is created by the CRI of the outer cluster, i.e., owned by the initial user namespace; this also cannot happen in the Docker Compose mode, where the network namespace is created inside the RootlessKit user namespace. Fix issue 396 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For a privileged pod, runc has to mount a fresh read-write sysfs, which the kernel allows in a mount namespace owned by a user namespace only when the mount namespace already contains a fully-visible read-write sysfs instance. The /sys of the node pods is mounted read-only by the CRI of the outer cluster, and the node pods cannot mount a read-write instance by themselves, so it has to be provided by the host, at /run/usernetes/sysfs, and mounted into the node pods via a hostPath volume. The parent directory is mounted instead of the sysfs itself, as sysfs does not support the idmapped mounts that the kubelet uses for the volumes of UserNS-enabled pods.
This issue does not occur in the Docker Compose mode: there the node container runs with
privileged: true(which is safe under Rootless Docker, as it only grants the capabilities that are namespaced in the RootlessKit user namespace), so Docker mounts its /sys read-write, which already satisfies the visibility requirement. A UserNS-enabled Kubernetes pod cannot be privileged, and the CRI has no option for mounting /sys of an unprivileged pod read-write (unlikecgroup_writablefor cgroups).Privileged pods with hostNetwork can be created even without this fix, although their /sys is a read-only bind mount of the /sys of the node pod, as mounting a fresh sysfs requires the network namespace to be owned by the user namespace. The network namespace of a node pod is created by the CRI of the outer cluster, i.e., owned by the initial user namespace; this also cannot happen in the Docker Compose mode, where the network namespace is created inside the RootlessKit user namespace.
Fix #396