ComputeDomain kubelet plugin: kubelet restart may permanently delete resource slice

[jgehrcke]

Before:

$ kubectl get resourceslices.resource.k8s.io 
NAME                                                  NODE                  DRIVER                      POOL                  AGE
gb-nvl-043-bianca-1-compute-domain.nvidia.com-cxknf   gb-nvl-043-bianca-1   compute-domain.nvidia.com   gb-nvl-043-bianca-1   12h
gb-nvl-043-bianca-2-compute-domain.nvidia.com-spqlr   gb-nvl-043-bianca-2   compute-domain.nvidia.com   gb-nvl-043-bianca-2   27m
gb-nvl-043-bianca-3-compute-domain.nvidia.com-99rtp   gb-nvl-043-bianca-3   compute-domain.nvidia.com   gb-nvl-043-bianca-3   12h
gb-nvl-043-bianca-4-compute-domain.nvidia.com-hzbqb   gb-nvl-043-bianca-4   compute-domain.nvidia.com   gb-nvl-043-bianca-4   25m
gb-nvl-043-bianca09-compute-domain.nvidia.com-tctnx   gb-nvl-043-bianca09   compute-domain.nvidia.com   gb-nvl-043-bianca09   12h

Then, on bianca-2:

$ sudo systemctl stop kubelet.service
$ sudo systemctl start kubelet.service

After:

$ kubectl get resourceslices.resource.k8s.io 
NAME                                                  NODE                  DRIVER                      POOL                  AGE
gb-nvl-043-bianca-1-compute-domain.nvidia.com-cxknf   gb-nvl-043-bianca-1   compute-domain.nvidia.com   gb-nvl-043-bianca-1   12h
gb-nvl-043-bianca-3-compute-domain.nvidia.com-99rtp   gb-nvl-043-bianca-3   compute-domain.nvidia.com   gb-nvl-043-bianca-3   12h
gb-nvl-043-bianca-4-compute-domain.nvidia.com-hzbqb   gb-nvl-043-bianca-4   compute-domain.nvidia.com   gb-nvl-043-bianca-4   26m
gb-nvl-043-bianca09-compute-domain.nvidia.com-tctnx   gb-nvl-043-bianca09   compute-domain.nvidia.com   gb-nvl-043-bianca09   12h

[jgehrcke]

Kevin suggests updating k8s.io/dynamic-resource-allocation:

https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/9e7def9940baa8e5888bc08caaadd11348d39fa4/go.mod#L21C9-L21C36

https://github.com/kubernetes/dynamic-resource-allocation/releases/tag/v0.33.0 was released two days ago.

We have a pretty decent repro, and can use that to find out if that bump will help.

[jgehrcke]

As before, the kubelet removes all ResourceSlices for its node at startup, then
DRA drivers recreate them if (and only if) they start up again. This ensures
that there are no orphaned ResourceSlices when a driver gets removed while the
kubelet was down.

from kubernetes/dynamic-resource-allocation@4322603

[jgehrcke]

I could not reproduce on Starwars, with 25.3.0-rc.2.

$ kubectl get resourceslice 
NAME                                                   NODE                   DRIVER                      POOL                   AGE
sc-starwars-mab5-b00-compute-domain.nvidia.com-lcw72   sc-starwars-mab5-b00   compute-domain.nvidia.com   sc-starwars-mab5-b00   5m37s
sc-starwars-mab6-b00-compute-domain.nvidia.com-fcz8t   sc-starwars-mab6-b00   compute-domain.nvidia.com   sc-starwars-mab6-b00   5m37s
sc-starwars-mab7-b00-compute-domain.nvidia.com-cmrv6   sc-starwars-mab7-b00   compute-domain.nvidia.com   sc-starwars-mab7-b00   5m37s
sc-starwars-mab8-b00-compute-domain.nvidia.com-6wcss   sc-starwars-mab8-b00   compute-domain.nvidia.com   sc-starwars-mab8-b00   5m37s

I then restarted kubelet on sc-starwars-mab6-b00 and right after that saw the slice disappear:

$ date; kubectl get resourceslice 
Wed May  7 03:06:16 PM UTC 2025
NAME                                                   NODE                   DRIVER                      POOL                   AGE
sc-starwars-mab5-b00-compute-domain.nvidia.com-lcw72   sc-starwars-mab5-b00   compute-domain.nvidia.com   sc-starwars-mab5-b00   10m
sc-starwars-mab7-b00-compute-domain.nvidia.com-cmrv6   sc-starwars-mab7-b00   compute-domain.nvidia.com   sc-starwars-mab7-b00   10m
sc-starwars-mab8-b00-compute-domain.nvidia.com-6wcss   sc-starwars-mab8-b00   compute-domain.nvidia.com   sc-starwars-mab8-b00   10m

Just a few seconds later the slice appeared again:

$ date; kubectl get resourceslice 
Wed May  7 03:06:40 PM UTC 2025
NAME                                                   NODE                   DRIVER                      POOL                   AGE
sc-starwars-mab5-b00-compute-domain.nvidia.com-lcw72   sc-starwars-mab5-b00   compute-domain.nvidia.com   sc-starwars-mab5-b00   10m
sc-starwars-mab6-b00-compute-domain.nvidia.com-j4qjc   sc-starwars-mab6-b00   compute-domain.nvidia.com   sc-starwars-mab6-b00   5s
sc-starwars-mab7-b00-compute-domain.nvidia.com-cmrv6   sc-starwars-mab7-b00   compute-domain.nvidia.com   sc-starwars-mab7-b00   10m
sc-starwars-mab8-b00-compute-domain.nvidia.com-6wcss   sc-starwars-mab8-b00   compute-domain.nvidia.com   sc-starwars-mab8-b00   10m

I would need to go back to the other environment to see what really caused the problem (or if the slice over there maybe re-appeared after a while by itself? I remember vaguely that @klueska and I looked together and the slice did not re-appear even after a while -- but memory fades away).

[shivamerla]

@jgehrcke do you think this can still happen or addressed with the recent versions?

[jgehrcke]

It's likely that either this never was an issue or it's not an issue anymore.