DRA: requeue can pair a Workload's newest object with stale preprocessed TotalRequests

[thc1006]

/kind bug
/area dra

What happened:

When a DRA-backed Workload is updated while it is inflight in the scheduler — popped for a scheduling attempt but not yet admitted — the ClusterQueue drops that update. PushOrUpdate returns early for inflight workloads, on the assumption that the requeue path will pick up the latest version:

// RequeueWorkload will handle placement with the latest version.
if c.workloads.HasInflight(key) {
    return
}

(pkg/cache/queue/cluster_queue.go:365-367)

The requeue path does fetch the newest object from the cache (manager.go:765), but for DRA workloads it then deliberately preserves the previously preprocessed DRA charge:

info.Update(log, &w, workload.WithPreserveTotalRequests())

(pkg/cache/queue/manager.go:781)

Info.Update sets Obj to the newest generation, but skips rebuilding TotalRequests when preserveTotalRequests is set (pkg/workload/workload.go:377). The scheduling hash is then recomputed from the mix — spec, priority and count come from the new Obj, while effectiveRequests/effectiveCount come from the stale TotalRequests:

i.SchedulingHash = computeSchedulingHash(log, i.Obj, i.TotalRequests)

(pkg/workload/workload.go:351, computeSchedulingHash at :389)

So generation G+1's object gets re-queued carrying generation G's DRA TotalRequests, and its scheduling-equivalence hash is derived from that mismatch.

What you expected to happen:

A re-queued Workload should be scheduled against a TotalRequests that reflects its current object — either by carrying the update that arrived while it was inflight, or by re-deriving the DRA charge before the next attempt — so that neither the fit decision nor the equivalence hash is computed from a spec/charge mismatch.

How to reproduce it (as minimally and precisely as possible):

Found by code inspection against main (6d6df4c58e). The window is:

1. A DRA-backed Workload (generation G) is popped for scheduling and becomes inflight, carrying TotalRequests(G).
2. Its effective footprint changes while inflight — for example a LimitRange default surfaces or raises an extended-resource request (cf. the AdjustResources timing discussed in DRA: an extended resource that only appears once Kueue has adjusted the Workload skips preprocessing #14393).
3. PushOrUpdate drops the fresh charge because the Workload is inflight (cluster_queue.go:366).
4. The scheduling attempt does not fit; RequeueWorkload re-queues the newest object (G+1) but with WithPreserveTotalRequests(), keeping TotalRequests(G) (manager.go:781).
5. The next scheduling cycle evaluates Obj(G+1) against TotalRequests(G), and — with SchedulingEquivalenceHashing enabled — an equivalence hash computed from that mismatch.

Anything else we need to know?:

The window is narrow and self-correcting: once the Workload leaves inflight, the next controller push goes through draRequestsChanged (cluster_queue.go:384) and refreshes TotalRequests before the following attempt, and the hash symptom additionally requires the SchedulingEquivalenceHashing feature. The worst effect I can see is a one-cycle over- or under-admission against the mismatched charge, so this is a latent inconsistency rather than a reliably reproducible admission bug.

This is distinct from the reconciler-side DRA-queueing issues #13930 (a requeue-after-backoff losing its preprocessed DRA resources) and #14035 (a single queueing point for DRA preprocessing), which are about the reconciler calling AddOrUpdateWorkload. This one is the scheduler manager's RequeueWorkload preserve-path, which deliberately keeps the DRA requests but pairs them with a newer object.

Possible directions, none obviously best:

• capture the update that arrived while inflight and prefer it at requeue over the preserved snapshot;
• skip the preserve when the object generation changed;
• re-run DRA preprocessing on the requeue path instead of preserving a stale snapshot.

Environment:

• Found by code inspection; not cluster-reproduced.
• Kueue version: main @ 6d6df4c58e.
• Kubernetes version / cloud / OS / kernel: n/a (static analysis).

---

This issue was investigated and written in part with the assistance of generative AI. I have verified the cited code paths and line references myself.

[Anjali-Chauhan1]

/assign