Rootful mode - node to node networking does not work

[vsoch]

Hi @AkihiroSuda - I was testing GPU with usernetes, and got fairly far but hit two erroneous cases:

• I can't access the GPU devices with rootless mode (see this issue)
• When I run docker with rootfull (confirmed by a startup message) none of the usernetes networking works. Meaning I can shell into a node and I can't contact any ip address on another node. Pod to pod if they are on the same node works.
• When I remove the need for the GPU and run the same workflow with CPU and rootless, everything works again.

I don't think it's in scope for you to help with getting GPUs working with rootless, but I'm hoping you might have insight for why networking stops working when it's rootful. From the readme it sounds like it should work?

Thanks!

[AkihiroSuda]

Is this issue reproducible on GHA?

[vsoch]

Probably not, unless you are able and willing to pump out money for custom runners with GPUs! 😆

I can barely get V100s on the clouds... can't imagine what regular use in CI would require.

[AkihiroSuda]

"Rootful mode - node to node networking does not work" does not seem relevant to GPUs?

[vsoch]

Good idea! Is there a way to do an equivalent of a restart with lima? I can install docker (rootful, skipping the dockerd-rootless installer tool), and add the user to the docker group, and normally then I need a login / out to get it working. In this case I'm getting a permissions error on the socket (because I haven't).

https://github.com/researchapps/usernetes/actions/runs/13460718730/job/37615121293

[vsoch]

Just found this - testing now!

[vsoch]

Yes! It works with that base! (This is the confirmation of rootful, as you know):

And the test fails:

I'll open a PR with this added test.

[vsoch]

All set: #366

Thanks for the help @AkihiroSuda 🙏

[vsoch]

To give you an update @AkihiroSuda - I've spent about 48 hours on the rootless case, and I've actually gotten it working several times with a strategy that uses cdi from the host. The problem is consistency, and all the manual tweaks / customizations that are required. For example, tonight I've brought up a few clusters per hour, and I'll get it working, try to harden the setup, but then when I bring up (what I deem to be) "the same" again, I get a slightly different error. I'll even see cases where it runs once, and then there is a containerd error about permissions. I'm not sure if you have experience about what might be causing that? That's the error that seems to be unsolvable in the sense that once I see it, there is no way to fix it and go back to a working state.

I'm probably not going to work on this over the weekend because I'm a bit behind on everything else that I should have been doing for the last few days. But if you are interested, the VM build, customizations, and setup branch is here. For safe keeping, this is the error that ultimately happens:

  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               116s  default-scheduler  Successfully assigned gpu-operator/nvidia-device-plugin-daemonset-hg6kz to u7s-flux-001
  Warning  FailedCreatePodSandBox  116s  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox "c3594993a3cfcfcee487887ac9b62f601a4da29985385691aa19f8c81c86c661": failed to create containerd task: failed to create shim task: failed to mount rootfs component: operation not permitted

I tried mounting a tmpfs there, and various attempts to cleanup and restart, no fix. What seems to be working OK is getting the container built from the host and being able to run nvidia-smi to see devices, but what isn't clear to me is what commands I should run with nvidia-ctk in the container to properly configure it with the gpu-operator and then have GPUs that are seen by the container. I've had this work a few times (and then fail) so there is something happening w.r.t state. I think the million dollar questions are:

1. What is going on with containerd permissions (and why)
2. What is the ideal way to expose the devices to containerd inside the usernetes node

[AkihiroSuda]

Found a workaround: execute ethtool --offload eth0 tx-checksum-ip-generic off in usernetes-node-1 container

Originally posted by @AkihiroSuda in #366 (comment)

---

The nvidia stuff is irrelevant to "Rootful mode - node to node networking does not work", and should be discussed in a separate issue.
I saw you already managed to get it work, though https://vsoch.github.io/2025/rootless-usernetes-gpu/

[vsoch]

Yes! Apologies for that. It was only relevant in that the NVIDIA GPUs / device plugins install easily with rootful mode, and helped me discover the bug here.