Description
designs/dra-scheduling.md states the RBAC requirement for enabling DRA as:
RBAC: operating the gate requires the Karpenter controller to have get;list;watch on resource.k8s.io resourceslices. Distributions enabling DRA must grant this (the provisioning path already lists slices; this adds the watch verb).
I granted exactly that and set IGNORE_DRA_REQUESTS=false. Both controller replicas then went into CrashLoopBackOff and stayed there — cluster-wide loss of provisioning, disruption and drift, not a degraded DRA path:
{"level":"ERROR","logger":"controller.controller-runtime.cache.UnhandledError","message":"Failed to watch",
"type":"*v1.ResourceClaim","error":"failed to list *v1.ResourceClaim: resourceclaims.resource.k8s.io is
forbidden: User \"system:serviceaccount:kube-system:karpenter\" cannot list resource \"resourceclaims\"
in API group \"resource.k8s.io\" at the cluster scope"}
...
panic: Timeout: failed waiting for *v1.ResourceClaim Informer to sync
github.com/samber/lo.must(...)
sigs.k8s.io/karpenter/pkg/controllers/dynamicresources/deviceallocation.(*Controller).Hydrate.func1()
sigs.k8s.io/karpenter@v1.14.0/pkg/controllers/dynamicresources/deviceallocation/controller.go:139
sigs.k8s.io/karpenter/pkg/controllers/dynamicresources/deviceallocation.(*Controller).Hydrate(...)
sigs.k8s.io/karpenter@v1.14.0/pkg/controllers/dynamicresources/deviceallocation/controller.go:136
sigs.k8s.io/karpenter/pkg/controllers/dynamicresources/deviceallocation.(*Controller).Register.func1(...)
sigs.k8s.io/karpenter@v1.14.0/pkg/controllers/dynamicresources/deviceallocation/controller.go:301
dynamicresources.deviceallocation keeps its own ResourceClaim informer (the design doc mentions the controller as the source of allocated devices, but not that it needs its own permission). Because Hydrate uses lo.Must, a permission gap is a hard startup panic rather than a degraded mode.
Granting resourceclaims fixes it. I also granted deviceclasses in the same rule pre-emptively, since claim validation resolves DeviceClass references through the controller-runtime client:
2a. DeviceClass Resolution: The DeviceClassName is resolved via the API server.
kubeClient: the controller-runtime client, used to resolve DeviceClass references during claim validation.
which is another cache-backed informer.
What I'd suggest (either would have prevented this)
- Update the RBAC paragraph in
designs/dra-scheduling.md to list every resource.k8s.io resource the controller watches — at minimum resourceslices and resourceclaims, plus deviceclasses if the client caches them.
- Fail with a clear, actionable error instead of a panic when a DRA informer cannot sync because of RBAC — e.g. surface the underlying
forbidden error and name the missing permission.
- Consider having the upstream chart grant these automatically when DRA is enabled, so distributions can't get it half-right.
Point 1 alone is probably enough; point 2 would turn a cluster-wide outage into a legible startup failure.
Versions
- karpenter v1.14.0 (commit 85eeae8), via karpenter-provider-aws v1.14.0
- Kubernetes v1.36.2-eks
IGNORE_DRA_REQUESTS=false, nvidia-dra-driver-gpu v25.12.0
Description
designs/dra-scheduling.mdstates the RBAC requirement for enabling DRA as:I granted exactly that and set
IGNORE_DRA_REQUESTS=false. Both controller replicas then went intoCrashLoopBackOffand stayed there — cluster-wide loss of provisioning, disruption and drift, not a degraded DRA path:dynamicresources.deviceallocationkeeps its ownResourceClaiminformer (the design doc mentions the controller as the source of allocated devices, but not that it needs its own permission). BecauseHydrateuseslo.Must, a permission gap is a hard startup panic rather than a degraded mode.Granting
resourceclaimsfixes it. I also granteddeviceclassesin the same rule pre-emptively, since claim validation resolvesDeviceClassreferences through the controller-runtime client:which is another cache-backed informer.
What I'd suggest (either would have prevented this)
designs/dra-scheduling.mdto list everyresource.k8s.ioresource the controller watches — at minimumresourceslicesandresourceclaims, plusdeviceclassesif the client caches them.forbiddenerror and name the missing permission.Point 1 alone is probably enough; point 2 would turn a cluster-wide outage into a legible startup failure.
Versions
IGNORE_DRA_REQUESTS=false, nvidia-dra-driver-gpu v25.12.0