Proposal: Opaque config in ResourceClaim status instead of Individual Mode

[pravk03]

Note: This is an exploratory proposal. This direction was @johnbelamaric's idea, which came up during a discussion about scalability concerns. Wanted to get general feedback on this direction before exploring it further.

Background

The DRA CPU driver currently has 2 modes exposed via the --cpu-device-mode flag:

1. Grouped Mode (--cpu-device-mode=grouped): Exposes CPUs grouped by topology boundaries (e.g., NUMA nodes, sockets). The driver remains in control of selecting which specific CPU cores within the boundaries of the selected device are allocated to the container.
2. Individual Mode (--cpu-device-mode=individual): Exposes every CPU core as an distinct device in the ResourceSlice. The main benefit this provides it allows users, or external schedulers to perform fine-grained selection of specific CPU cores at the claim level.

Individual mode while it technically works with default kube-scheduler the scalability challenges prevent it from being used broadly. For all practical purposes is primarily meant for external scheduler use-cases where there is a need to select exact CPUs. This mode presents the below scalability challenges

1. Kube-scheduler allocation logic does not scale well with 100's of devices per node/ResourceSlice. This performance bottleneck occurs when a claim requests a general count of devices (e.g., "give me any 4 CPUs") rather than requesting specific device instances (Eg: "give me CPUS 0-3"). This forces the kube-scheduler's allocation algorithm to search for and choose available devices and this performs poorly at scale. This is the main reason why individual mode is recommended only with external schedulers.
2. Requesting a large number of devices in a claim easily hits the DRA default claim limit of 32 devices (Issue #5717). We need to split up requests into multiple claims or increase the default value.
3. Large machines easily hit limits on the maximum number of devices stored per ResourceSlice (Issue #5718). We need to split into multiple ResourcesSlices.
4. More etcd storage.

Additionally, maintaining the two modes creates two separate code paths with very little overlap. #112 has more discussion and it was agreed that individual mode can be split as a seperate driver.

Alternate Proposal

We could use grouped mode with the below changes as an alternative to Individual mode to achieve fine grain allocation:

1. Along with currently supported groupings (NUMA node, socket), we introduce a new configuration option: group-by=machine. In this mode, the DRA driver exposes all CPUs on the node as a single device as a consumable capacity.

Example ResourceSlice:

devices:
    - name: cpudevmachine
      allowMultipleAllocations: true
      capacity:
        dra.cpu/cpu:
          value: "256"

2. When group-by=machine is configured, external schedulers can inject specific CPU requirements into the claim's status allocation configuration at bind/allocation time using status.devices.config.opaque. The DRA driver parses this and enforces the CPUMask instead of running the allocation logic.

Example Claim.

  apiVersion: resource.k8s.io/v1
  kind: ResourceClaim
  metadata:
    name: claim-cpu-10
  spec:
    devices:
      requests:
      - name: req-cpu
        exactly:
          deviceClassName: dra.cpu
          capacity:
            requests:
              dra.cpu/cpu: "10"
  status:
    allocation:
      devices:
        . . .
        config:  # Added by external scheduler during binding
        - source: FromClaim
          requests:
          - req-cpu
          opaque:
            driver: dra.cpu
            parameters:
              cpuset: "2-11"

• DRA driver behavior in group-by=machine:
  • When opaque is not specified: Fall back to DRA driver allocating CPUs based on claim request [Edit]: With machine grouping we always require opaque config passed. More discussion in the implementation PR.
  • When opaque is specified: Validate and allocate the specified cores directly:
    • Validation: The driver verifies that the custom cpuset is valid for the host machine and is currently allocatable:
      • The number of CPUs in cpuset must exactly match what is requested in the claim (dra.cpu/cpu: "10").
      • cpuset is valid for the node.
      • cpuset is not already reserved using the driver's --reserved-cpus configuration flag.
      • cpuset is not already already allocated to any other active claims.
    • Error Handling: If any of the above validation fails, the driver returns a failure immediately in Kubelet's PrepareResourceClaims hook, causing pod startup to fail.

Proposal: Can we remove Individual mode and use the mechanism described above for fine-grained CPU allocation ?

Pros

• No scalability concerns stemming from device count.
• Simplifies driver logic by avoiding having to maintain 2 code paths.

Cons

• Without individual mode, we cannot request for specific CPUs through claims using the default kube-scheduler.
• If multiple claims have overlapping CPUs, validation failure occurs on the node during the PrepareResources phase (after the pod is scheduled), leaving the pod stuck on the node (This scenario likely represents a bug in the external scheduler code though).
• (Maybe more that I havn't thought through)

[pravk03]

cc @johnbelamaric @catblade @dchen1107 @yujuhong @wickberg @ffromani

[catblade]

The 32-core limit is causing real headaches, especially with how resourcelaims must be constructed by the scheduler. This was the most expedient (and nice and simple) alternative.

[ffromani]

cc @johnbelamaric @catblade @dchen1107 @yujuhong @wickberg @ffromani

thanks for this proposal!

Disclosure: I'm mostly (and I mostly represent) a grouped mode user and userbase, but clearly I want to support the usecases individual mode enables, albeit so far I have had very sporadic need for them.

That said, I like this approach, a lot. I also glanced at the POC and it's refreshingly simple and clean.

Let's discuss this and let's see if @catblade and @wickberg are positive this can help their usecases. Perhaps talk about this in the next WG-device-management mtg?

If this direction seems promising and we can get a preliminar design-level validation, we can change our schedule and the next steps accordingly.

Pending discussion and validation, this is how it could look like:

1. add the POC to 0.2.0
2. postpone [BLOCKER][GATED] 0.3.0: switch defaults, deprecate grouped mode #152 and Split the driver and retain only the individual mode #112 to v0.3.0 - possibly reconsidering the decision to split, if this approach solves the individual mode usecases we should deprecate + remove it and just ("just") rename this driver to cpumem and implement Absorb the memory allocation feature #36
3. update PLANNING: 2026 general roadmap #153 accordingly

I think the only must-have is wait to go forward with 0.2.0 because 0.2.0 is the first step which commits to the split. Therefore much harder to pivot back once that lands.
But the timeline for 0.2.0 was intentionally flexible so waiting a couple weeks to discuss this option is a very fair price.

[pravk03]

Thanks @ffromani. Agree with the plan outlined above.

We can include this in v0.2.0 so that we can get more feedback. We can bring this up in WG/SIG meetings once we confirm that this new approach works for current individual mode use cases. If things work as expected, we can have it as a goal for v0.3.0 to mark individual mode as deprecated.

+1 to also renaming the existing repository (when we absorb memory support) instead of creating a new one if things work out well with this proposal.

[catblade]

I have gone ahead and gotten our repo working against @pravk03 's POC. Works fine from my testing. I think this is better, as it will allow enumeration of a larger number of devices.

[ffromani]

I'm very happy to report #165 merged and we are very interested in feedback and to learn if individual mode can be deprecated in favor of group-by=machine

[ffromani]

/assign @pravk03

reflecting the reality Praveen is driving this work