Repository navigation
Commit b0ca1ff
committed
[SPARK-59712] Apply Kueue
### What changes were proposed in this pull request?
Apply the `nodeLabels` and the `tolerations` of the Kueue `ResourceFlavor`s which an admitted
`Workload` was assigned to the pods the operator creates for it, like Kueue built-in integrations
do in `podset.FromAssignment` and `podset.Merge` (Kueue v0.19.4). This wires up
`KueueWorkloadUtils.resolvePodSetFlavors`, which SPARK-59654 added without a caller.
`holdForAdmission` resolves the flavors and hands them to the context through
`BaseContext.setKueuePodSetFlavors`, which applies them where each role gets its pods from.
| Resource | Where the flavors are applied |
|---|---|
| `SparkCluster` | The master and worker `StatefulSet` pod templates, in place |
| `SparkApplication` | The driver and executor pod templates of a copy of the resource, since the driver creates the executors. The copy keeps the pod sets of the `Workload`, and their hash, unchanged |
| Case | Behavior |
|---|---|
| The driver or the master exists already | The flavors are resolved again, since rebuilding the resources without them would remove what server-side apply owns |
| A node label conflicts with the node selector of the pods | Permanent, like `ErrInvalidPodSetUpdate` in Kueue: the quota is released and the resource fails with `SchedulingFailure`. A failed release is retried rather than reported |
| The `ResourceFlavor`s cannot be read | The resource is held and the read is retried with the new `KueueResourceFlavorReadFailed` warning, rather than applying no flavors |
| The flavors carry neither node labels nor tolerations | Left out, so the stock `default-flavor` costs nothing |
Topology Aware Scheduling and the `podSetUpdates` of admission checks remain out of scope.
### Why are the changes needed?
A `ResourceFlavor` usually stands for a node group, and applying Kueue's assignment to the pods is
the job of the integration. Without it the operator charges the quota of one node group while the
pods land on another, pods admitted to a tainted node group stay `Pending` while holding that
quota, and flavor fungibility does not work for Spark resources.
### Does this PR introduce _any_ user-facing change?
No, Kueue feature is not released yet.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #864 from dongjoon-hyun/SPARK-59712.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>ResourceFlavor node labels and tolerations to Spark pods1 parent c33fa30 commit b0ca1ff
19 files changed
Lines changed: 1281 additions & 82 deletions
File tree
- docs
- spark-operator/src
- main/java/org/apache/spark/k8s/operator
- context
- kueue
- reconciler
- reconcilesteps
- utils
- test/java/org/apache/spark/k8s/operator
- context
- kueue
- reconciler
- reconcilesteps
- tests/e2e/kueue
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| 80 | + | |
80 | 81 | | |
81 | 82 | | |
82 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
624 | 624 | | |
625 | 625 | | |
626 | 626 | | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
627 | 639 | | |
628 | 640 | | |
629 | 641 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| 29 | + | |
27 | 30 | | |
28 | 31 | | |
29 | 32 | | |
| |||
35 | 38 | | |
36 | 39 | | |
37 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
38 | 44 | | |
39 | 45 | | |
40 | 46 | | |
| |||
51 | 57 | | |
52 | 58 | | |
53 | 59 | | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
54 | 80 | | |
55 | 81 | | |
56 | 82 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
179 | 179 | | |
180 | 180 | | |
181 | 181 | | |
182 | | - | |
| 182 | + | |
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | 186 | | |
187 | 187 | | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
188 | 201 | | |
189 | 202 | | |
190 | 203 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
65 | 67 | | |
66 | 68 | | |
67 | 69 | | |
| 70 | + | |
68 | 71 | | |
69 | 72 | | |
70 | 73 | | |
71 | 74 | | |
72 | 75 | | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
73 | 104 | | |
74 | 105 | | |
75 | 106 | | |
| |||
0 commit comments