DRA KEP 5004: Handle extended resource requests via DRA Driver

[jmdeal]

Overview

KEP	Graduation Status
5004	GA (1.37)

KEP 5004 enables users to request DRA resources through the extended resource API. A device class is associated with an extended resource (either explicitly or though an implicit device class identifier), enabling pods to request resources matching that device class' constraints.

For example, the following device class definition surfaces GPUs as an extended resource:

apiVersion: resource.k8s.io/v1beta1
kind: DeviceClass
metadata:
  name: gpu.example.com
spec:
  selectors:
  - cel:
      expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type
        == 'gpu'
  extendedResourceName: example.com/gpu

Note that the same resource name, in this case example.com/gpu may be surfaced by a non-DRA driver. In this scenario, a pod may schedule via either resource.

Karpenter Integration

While the scheduler does create a ResourceClaim to track allocation metadata for the pod, this claim is not created until the scheduler's pre-bind phase. Since Karpenter reacts to pending pods, we can not rely on plumbing this claim through our scheduler. We'll instead need to create virtual claims for each pod requesting these extended resources.

An additional consideration is when an instance type has offering that advertise both a classic extended resource and DRA resources that can be mapped to an extended resource of the same name. This is a motivating use-case of this feature, as it enables a transition to DRA without requiring application owners to update their resource requests. This poses a challenge for Karpenter's NodeClaim initialization check — we need to model on the NodeClaim that we need one of the two types of resources to be published by the resulting node, but it differs depending on the offering / launch configuration that is chosen by the cloudprovider implementation. We will likely need to extend the NodeClaim interface to model this.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• Comments asking for updates generate extra noise for issue followers and do not help maintainers prioritize issues. If you think the issue is higher priority than it is labelled, please instead add your user story.
• If you are interested in working on this issue or have submitted a pull request, please assign yourself

[yindia]

@jmdeal I'd like to pick this up, so assigning myself.

I spent some time sketching out how the Karpenter side could work and wrote it up as an RFC. Quick summary in case the approach is worth discussing before I open the PR.

The core problem is that these pods don't carry pod.spec.resourceClaims. The claim gets created by kube-scheduler at PreBind, which is well after Karpenter has already looked at the pod, so our current DRA path never sees them. Today the request just falls through to the scalar path and gets checked against instance-type Capacity, which is empty on a DRA-only offering, so the pod looks unschedulable.

What I'm proposing is to build an extendedResourceName -> DeviceClass map once per loop (covering both the explicit field and the implicit deviceclass.resource.kubernetes.io/<class-name> name, with the KEP tie-break), then synthesize an in-memory ResourceClaim for the pod and hand it to the existing allocator. For ExactCount that seems to need no allocator changes, which was the part I most wanted to confirm.

The trickier case is the one called out in the description: an instance type advertising the same name as both a device-plugin resource and a DRA-backed one. Since the allocator already evaluates a NodeClaim as a superposition of candidate instance types, I think we can route through DRA only for offerings that don't advertise the name in Capacity and let the scalar path handle the rest.

For init I'd rather not add a NodeClaim field yet. On the launched offering we either keep the name in spec.resources.requests (device-plugin path) or record the driver in the DRA drivers annotation (DRA path), so init waits on whichever one actually got used. A new field only really becomes necessary if the offering choice gets deferred to the cloud provider at launch, which I flagged as an open question.

I was planning to scope the first PR to explicit extendedResourceName plus ExactCount, behind the existing DRA gate, with implicit names and FirstAvailable as follow-ups. Full detail (the wire-in points, edge cases, and alternatives considered) is in the RFC I'll open against designs/.

Does that line up with where you all want DRA to go? Happy to open it once there's rough agreement.