Skip to content

DRA: controller panics on startup without resourceclaims RBAC, but only resourceslices is documented #3249

Description

@Poil

Description

designs/dra-scheduling.md states the RBAC requirement for enabling DRA as:

RBAC: operating the gate requires the Karpenter controller to have get;list;watch on resource.k8s.io resourceslices. Distributions enabling DRA must grant this (the provisioning path already lists slices; this adds the watch verb).

I granted exactly that and set IGNORE_DRA_REQUESTS=false. Both controller replicas then went into CrashLoopBackOff and stayed there — cluster-wide loss of provisioning, disruption and drift, not a degraded DRA path:

{"level":"ERROR","logger":"controller.controller-runtime.cache.UnhandledError","message":"Failed to watch",
 "type":"*v1.ResourceClaim","error":"failed to list *v1.ResourceClaim: resourceclaims.resource.k8s.io is
 forbidden: User \"system:serviceaccount:kube-system:karpenter\" cannot list resource \"resourceclaims\"
 in API group \"resource.k8s.io\" at the cluster scope"}
...
panic: Timeout: failed waiting for *v1.ResourceClaim Informer to sync

github.com/samber/lo.must(...)
sigs.k8s.io/karpenter/pkg/controllers/dynamicresources/deviceallocation.(*Controller).Hydrate.func1()
	sigs.k8s.io/karpenter@v1.14.0/pkg/controllers/dynamicresources/deviceallocation/controller.go:139
sigs.k8s.io/karpenter/pkg/controllers/dynamicresources/deviceallocation.(*Controller).Hydrate(...)
	sigs.k8s.io/karpenter@v1.14.0/pkg/controllers/dynamicresources/deviceallocation/controller.go:136
sigs.k8s.io/karpenter/pkg/controllers/dynamicresources/deviceallocation.(*Controller).Register.func1(...)
	sigs.k8s.io/karpenter@v1.14.0/pkg/controllers/dynamicresources/deviceallocation/controller.go:301

dynamicresources.deviceallocation keeps its own ResourceClaim informer (the design doc mentions the controller as the source of allocated devices, but not that it needs its own permission). Because Hydrate uses lo.Must, a permission gap is a hard startup panic rather than a degraded mode.

Granting resourceclaims fixes it. I also granted deviceclasses in the same rule pre-emptively, since claim validation resolves DeviceClass references through the controller-runtime client:

2a. DeviceClass Resolution: The DeviceClassName is resolved via the API server.
kubeClient: the controller-runtime client, used to resolve DeviceClass references during claim validation.

which is another cache-backed informer.

What I'd suggest (either would have prevented this)

  1. Update the RBAC paragraph in designs/dra-scheduling.md to list every resource.k8s.io resource the controller watches — at minimum resourceslices and resourceclaims, plus deviceclasses if the client caches them.
  2. Fail with a clear, actionable error instead of a panic when a DRA informer cannot sync because of RBAC — e.g. surface the underlying forbidden error and name the missing permission.
  3. Consider having the upstream chart grant these automatically when DRA is enabled, so distributions can't get it half-right.

Point 1 alone is probably enough; point 2 would turn a cluster-wide outage into a legible startup failure.

Versions

  • karpenter v1.14.0 (commit 85eeae8), via karpenter-provider-aws v1.14.0
  • Kubernetes v1.36.2-eks
  • IGNORE_DRA_REQUESTS=false, nvidia-dra-driver-gpu v25.12.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions