study: investigate runtime enforcer visibility into gVisor/runsc containers

[dottorblaster]

We want to understand whether the enforcer can provide meaningful visibility into gVisor/runsc containers.

gVisor's Sentry intercepts guest syscalls in userspace, so sandboxed workloads don't issue "real" syscalls from the host's point of view. This breaks most assumptions our eBPF probes rely on. We need to quantify the gap before deciding if there's something to build.

Acceptance criteria

• Capability matrix: feature → { works / partial / blind } with one-line rationale.
• Root-cause notes for "blind" cases.
• Recommendation: doc gap, product gap, or "don't support".

[holyspectral]

My understanding is that gvisor/runsc runs in a micro VM with their own kernels, so we can't see those information with the current setup of runtime-enforcer.  I know falco supported it via trace_session of gvisor, but the feature has been deprecated. I also know other solutions deploy a micro agent as the container entrypoint to receive those information via ptrace.  All in all, this would need us to revisit the status quo to see what's available at the moment as this is highly dependent on what gvisor/runsc provides. One significant gap I can see is that gVisor's Sentry intercepts syscalls, which could be difficult to map back to the real file path if symlink is involved.

[Andreagit97]

Uhm, since we are not interested in syscall tracking, I'm wondering if the gvisor setup really requires big architectural changes for us. In the end, if I understand well, the Sentry (emulated Linux kernel) should really create the processes inside the real host kernel where we have our eBPF program. https://gvisor.dev/blog/2019/11/18/gvisor-security-basics-part-1/ Or maybe I'm wrong, and the process management is completely internal to Sentry, even if it seems strange

For sure, there are differences in the cgroup management. I quickly tried:

minikube start --driver=kvm2 --container-runtime=containerd
minikube addons enable gvisor

# 1. Deployed our runtime enforcer...

# 2. Deployed a test Ubuntu deployment with Gvisor enabled (see below)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-deployment-gvisor
  namespace: default
  labels:
    app: ubuntu-deployment-gvisor
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ubuntu-gvisor
      type: deployment
  template:
    metadata:
      labels:
        app: ubuntu-gvisor
        type: deployment
    spec:
      runtimeClassName: gvisor
      terminationGracePeriodSeconds: 1
      containers:
      - name: ubuntu
        image: ubuntu
        # We create a while loop to continually run sleep and ls.
        # This way we can detect two groups of processes in our test:
        # - process snapshot (bash)
        # - process events (sleep and ls).
        command: ["bash", "-c", "while true; do sleep 5; ls; done"]

This is the error from NRI

  {"time":"2026-04-20T12:20:07.48504881Z","level":"ERROR","msg":"failed to get cgroup ID from container: failed to get cgroup ID from path '/proc/1/root/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1bc160_884d_4e26_9a29_57497e462d5e.slice/cri-containerd-d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f.scope' for container 'ubuntu(d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f)': nameToHandle on /proc/1/root/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1bc160_884d_4e26_9a29_57497e462d5e.slice/cri-containerd-d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f.scope failed: no such file or directory. Runtime-enforcer has prevented the container 'ubuntu-deployment-gvisor-648bcf55f8-qptbl/ubuntu' from starting. To change this behavior, set agent.nriFailopen=true in the helm chart","component":"agent","component":"nri-handler","component":"nri-plugin","pod":{"name":"ubuntu-deployment-gvisor-648bcf55f8-qptbl","namespace":"default","uid":"bd1bc160-884d-4e26-9a29-57497e462d5e"},"container":{"name":"ubuntu","id":"d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f"},"reason":"failed to get cgroup ID from container","error":"failed to get cgroup ID from path '/proc/1/root/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1bc160_884d_4e26_9a29_57497e462d5e.slice/cri-containerd-d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f.scope' for container 'ubuntu(d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f)': nameToHandle on /proc/1/root/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1bc160_884d_4e26_9a29_57497e462d5e.slice/cri-containerd-d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f.scope failed: no such file or directory"}

The cgroup path /proc/1/root/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd1bc160_884d_4e26_9a29_57497e462d5e.slice/cri-containerd-d1fd62180601515f848608770e9b434073727ff28eda1d64efbe1dfa943d598f.scope is definitely wrong, it seems gvisor doesn't create a dedicated cgroup (systemd .scope) for each container...

Normal pod has cri-containerd-4c292f9655a05b3a54428c928b348533bc11d6a4d183e84b8d9245ff98063e72.scope

cgroup.controllers cpu.idle        cpuset.mems.effective              memory.current    memory.stat
cgroup.events  cpu.max         cri-containerd-34d8066da40b5fd0f5cb351e414250066b34106f23bb4748e50580363095ee1d.scope  memory.events    memory.swap.current
cgroup.freeze  cpu.max.burst        cri-containerd-4c292f9655a05b3a54428c928b348533bc11d6a4d183e84b8d9245ff98063e72.scope  memory.events.local  memory.swap.events
cgroup.kill  cpu.stat        hugetlb.2MB.current              memory.high    memory.swap.high
cgroup.max.depth cpu.stat.local        hugetlb.2MB.events              memory.low    memory.swap.max
cgroup.max.descendants cpu.weight        hugetlb.2MB.events.local              memory.max    memory.swap.peak
cgroup.procs  cpu.weight.nice        hugetlb.2MB.max               memory.min    pids.current
cgroup.stat  cpuset.cpus        hugetlb.2MB.numa_stat              memory.numa_stat    pids.events
cgroup.subtree_control cpuset.cpus.effective  hugetlb.2MB.rsvd.current              memory.oom.group    pids.max
cgroup.threads  cpuset.cpus.partition  hugetlb.2MB.rsvd.max              memory.peak    pids.peak
cgroup.type  cpuset.mems        io.stat                memory.reclaim

Gvisor pod cgroup

cgroup.controllers cgroup.procs  cpu.max   cpuset.cpus  hugetlb.2MB.events   io.stat        memory.max  memory.stat       pids.current
cgroup.events  cgroup.stat  cpu.max.burst  cpuset.cpus.effective hugetlb.2MB.events.local  memory.current       memory.min  memory.swap.current  pids.events
cgroup.freeze  cgroup.subtree_control cpu.stat  cpuset.cpus.partition hugetlb.2MB.max    memory.events        memory.numa_stat  memory.swap.events   pids.max
cgroup.kill  cgroup.threads  cpu.stat.local  cpuset.mems  hugetlb.2MB.numa_stat   memory.events.local  memory.oom.group  memory.swap.high     pids.peak
cgroup.max.depth cgroup.type  cpu.weight  cpuset.mems.effective hugetlb.2MB.rsvd.current  memory.high        memory.peak  memory.swap.max
cgroup.max.descendants cpu.idle  cpu.weight.nice  hugetlb.2MB.current hugetlb.2MB.rsvd.max   memory.low        memory.reclaim  memory.swap.peak

[holyspectral]

@Andreagit97 thanks for looking into it. Yes I think it's expected that we can see some events from NRI because the information is provided by high-level runtime. The problem is more about the visibility inside the container.

I did an experiment with a Deployment to run sleep 10000. I have ftrace enabled inside a minikube instance.

$ echo 1 > events/sched/sched_process_exec/enable
$ echo 1 > tracing_on
$ cat trace_pipe

Then I compared two types of workload with gvisor enabled/disabled.

• With gvisor enabled, I can't see anything about the container's processes except runsc: runsc.txt

• On the other hand, I can see sleep runs if gvisor is disabled: runc.txt

To make sure if runsc just wrapps the command somehow, I also tried to run while true; do sleep 1;echo waiting; done in the container with gvisor enabled. There are only few runsc events detected in host kernels (~10) compared to the real sleep processes (~120). In the runc case, I can see the process event once per second.

Not saying that it's completely impossible to support this with the current architecture, but it does seem like a limitation of our current approach, given that we rely on the host kernel to detect these activities.

[Andreagit97]

I see 👀 it would be interesting to understand how Gvisor models the processes internally, but this is something we can definitely take a look at when we will pick up the issue