[Bug]: k8s-mig-manager v0.14.0 observes nvidia.com/mig.config label but does not apply geometry on Talos

[TobiPeterG]

Describe the bug
When running standalone k8s-mig-manager on Talos Linux with preinstalled NVIDIA drivers/toolkit from Talos system extensions, v0.14.0 observes changes to the nvidia.com/mig.config node label but does not perform a real MIG geometry transition.

The underlying nvidia-mig-parted command works manually in the same pod. Downgrading the manager image to v0.12.3-ubuntu20.04 makes label-driven reconfiguration work.

To Reproduce
MIG mode was enabled on both A100s. The node initially had this applied profile:

nvidia.com/mig.config=all-1g.5gb
nvidia.com/mig.config.state=success

With v0.14.0, changing the label:

kubectl label node gx06 nvidia.com/mig.config=all-balanced --overwrite

updated the node label, but the hardware geometry did not change.

k8s-mig-manager logs stopped after:

Using custom config: /mig-parted-config/config.yaml
Waiting for change to 'nvidia.com/mig.config' label
Updating to MIG config: all-1g.5gb

No later logs such as applying the config or successfully updating appeared.

nvidia-smi -L still showed the old all-1g.5gb layout.

Manual assert for the requested config failed:

kubectl exec -n nvidia-mig-manager ds/nvidia-mig-manager -- nvidia-mig-parted -d assert -f /mig-parted-config/config.yaml -c all-balanced

Output included:

Not all GPUs match the specified config
Assertion failure: selected configuration not currently applied

However, manual apply from inside the same pod worked:

kubectl exec -n nvidia-mig-manager ds/nvidia-mig-manager -- nvidia-mig-parted -d apply -f /mig-parted-config/config.yaml -c all-balanced

Expected behavior
k8s-mig-manager v0.14.0 should apply the selected profile when nvidia.com/mig.config changes, then set:

nvidia.com/mig.config.state=success

Environment (please provide the following information):

• OS: Talos Linux v1.12.6
• Kubernetes: v1.34.6
• Node: gx06
• GPUs: 2x NVIDIA A100-PCIE-40GB
• NVIDIA driver/toolkit: provided by Talos system extensions (580.126.20)
• Kernel Version: 6.18.18-talos
• GPU Operator: not installed
• RuntimeClass: nvidia
• MIG Manager image that fails: nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.14.0
• MIG Manager image that works: nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.12.3-ubuntu20.04

Notes

This appears specific to the standalone k8s-mig-manager v0.14.0 controller path, not to nvidia-mig-parted itself. The same container could run nvidia-mig-parted apply manually.

One visible difference is that newer v0.14.0 uses the Go-native reconfigure path, while v0.12.3 uses the older shell-based reconfigure-mig.sh path.

[rajathagasthya]

@TobiPeterG Thanks for the detailed report. Talos is not an officially supported platform, but it should still work. Could you please share some additional details?

1. Complete pod logs of MIG Manager. Previous logs would be helpful too.

kubectl logs -n nvidia-mig-manager ds/nvidia-mig-manager
kubectl logs -n nvidia-mig-manager ds/nvidia-mig-manager --previous

2. Is WITH_SHUTDOWN_HOST_GPU_CLIENTS env var set in MIG Manager?
3. Can you check whether a D-Bus system socket exists on the host from inside the mig-manager pod?

kubectl exec -n nvidia-mig-manager ds/nvidia-mig-manager -- ls -la /host/run/dbus/system_bus_socket

The Go-native reconfigure looks for this socket to initialize a systemd D-Bus connection. We'd see an error if it was failing (or it might hang), so I want to rule it out.

[TobiPeterG]

@rajathagasthya

Thanks. I reproduced the issue with v0.14.0 while the node was cordoned:

Current image:

nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.14.0

Environment from the DaemonSet:

WITH_SHUTDOWN_HOST_GPU_CLIENTS=false
WITH_REBOOT=false
HOST_ROOT_MOUNT=/host
HOST_KUBELET_SYSTEMD_SERVICE=kubelet.service
HOST_MIG_MANAGER_STATE_FILE=/etc/systemd/system/nvidia-mig-manager.service.d/override.conf

The host D-Bus socket exists inside the pod:

srwxr-xr-x    1 0        0                0 Apr 28 12:12 /host/run/dbus/system_bus_socket

kubectl logs --previous returned:

previous terminated container "nvidia-mig-manager" in pod "nvidia-mig-manager-vzhbq" not found

because the pod did not restart.

Complete v0.14.0 logs after startup:

ERROR: init 250 result=11W0505 09:57:29.259125  152055 client_config.go:682] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2026-05-05T09:57:29Z" level=info msg="Using custom config: /mig-parted-config/config.yaml"
time="2026-05-05T09:57:29Z" level=info msg="Waiting for change to 'nvidia.com/mig.config' label"
time="2026-05-05T09:57:29Z" level=info msg="Updating to MIG config: all-7g.40gb"

At this point the manager did not log success/failure and did not return to waiting for another label change.

I then changed the label:

kubectl label node gx06 nvidia.com/mig.config=all-1g.5gb --overwrite

No additional logs appeared from v0.14.0.

The hardware layout remained at the previous profile:

GPU 0: NVIDIA A100-PCIE-40GB
  MIG 7g.40gb     Device  0
GPU 1: NVIDIA A100-PCIE-40GB
  MIG 7g.40gb     Device  0

Manual assert for the newly requested label failed as expected:

Not all GPUs match the specified config
Assertion failure: selected configuration not currently applied

The manager process wait channel while stuck was:

futex_do_wait

After this, I rolled back to:

nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.12.3-ubuntu20.04

I hope this helps :)

[Arc676]

I also observe this behavior and can confirm that the downgrade works (thanks @TobiPeterG for identifying this). In the environment I work in, we install the NVIDIA drivers via Talos system extension, so WITH_SHUTDOWN_HOST_GPU_CLIENTS and IS_HOST_DRIVER are both true.

The Go-native reconfigure looks for this socket to initialize a systemd D-Bus connection. We'd see an error if it was failing (or it might hang), so I want to rule it out.

I suspect this is indeed the issue. As specified in the Talos documentation, Talos does not run systemd. Running a debug container for the node shows that the DBus socket exists, but there is no traffic: running socat -v - UNIX-CONNECT:/host/run/dbus/system_bus_socket yields no output.

The code blocks relating to systemd and persisting the configuration on the host are likely unportable to Talos (no systemd and the OS is immutable). The assumption that the GPU services on the host can be restarted using systemd does not apply. The rest of the code should still work fine; there would just need to be a flag to skip the affected sections. I'm not entirely sure how this could be implemented in a vendor-agnostic manner; perhaps a flag --disable-systemd.

Environment:

• OS: Talos Linux v1.13.0
• Kubernetes: v1.35.1
• GPUs: NVIDIA H100 NVL
• NVIDIA driver/toolkit: provided by Talos system extensions (580.126.20)
• Kernel Version: 6.18.24-talos
• GPU Operator: Helm chart v25.10.1

---

Side note:

Talos is not an officially supported platform, but it should still work.

Since Talos is an immutable OS, the MIG partitioning binary cannot be copied to the host. The aforementioned chart forces WITH_SHUTDOWN_HOST_GPU_CLIENTS=IS_HOST_DRIVER, which means that mig-parted will try to copy the binary to the host and fail.

More generally, there would have to be a way to manually disable WITH_SHUTDOWN_HOST_GPU_CLIENTS because, as stated before, there is no systemd on Talos. All the code specific to restarting the NVIDIA services via systemd doesn't apply to Talos machines.