Skip to content

Commit b0ca1ff

Browse files
committed
[SPARK-59712] Apply Kueue ResourceFlavor node labels and tolerations to Spark pods
### What changes were proposed in this pull request? Apply the `nodeLabels` and the `tolerations` of the Kueue `ResourceFlavor`s which an admitted `Workload` was assigned to the pods the operator creates for it, like Kueue built-in integrations do in `podset.FromAssignment` and `podset.Merge` (Kueue v0.19.4). This wires up `KueueWorkloadUtils.resolvePodSetFlavors`, which SPARK-59654 added without a caller. `holdForAdmission` resolves the flavors and hands them to the context through `BaseContext.setKueuePodSetFlavors`, which applies them where each role gets its pods from. | Resource | Where the flavors are applied | |---|---| | `SparkCluster` | The master and worker `StatefulSet` pod templates, in place | | `SparkApplication` | The driver and executor pod templates of a copy of the resource, since the driver creates the executors. The copy keeps the pod sets of the `Workload`, and their hash, unchanged | | Case | Behavior | |---|---| | The driver or the master exists already | The flavors are resolved again, since rebuilding the resources without them would remove what server-side apply owns | | A node label conflicts with the node selector of the pods | Permanent, like `ErrInvalidPodSetUpdate` in Kueue: the quota is released and the resource fails with `SchedulingFailure`. A failed release is retried rather than reported | | The `ResourceFlavor`s cannot be read | The resource is held and the read is retried with the new `KueueResourceFlavorReadFailed` warning, rather than applying no flavors | | The flavors carry neither node labels nor tolerations | Left out, so the stock `default-flavor` costs nothing | Topology Aware Scheduling and the `podSetUpdates` of admission checks remain out of scope. ### Why are the changes needed? A `ResourceFlavor` usually stands for a node group, and applying Kueue's assignment to the pods is the job of the integration. Without it the operator charges the quota of one node group while the pods land on another, pods admitted to a tainted node group stay `Pending` while holding that quota, and flavor fungibility does not work for Spark resources. ### Does this PR introduce _any_ user-facing change? No, Kueue feature is not released yet. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 Closes #864 from dongjoon-hyun/SPARK-59712. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent c33fa30 commit b0ca1ff

19 files changed

Lines changed: 1281 additions & 82 deletions

‎docs/configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@ In addition, the operator publishes the following `Warning` events.
7777
| `CleanupError` | A cleanup throws, so the resource cannot finish deleting. |
7878
| `StatusUpdateFailed` | A status patch is rejected. Transport-level errors are skipped. |
7979
| `KueueAdmissionRequestFailed` | Creating, reading or deleting a stale Kueue `Workload` fails, or the driver or master cannot be read before the admission is requested. Transport-level errors are skipped and retried every 5 seconds, while a persistent failure is retried with the default interval. |
80+
| `KueueResourceFlavorReadFailed` | Reading the `ResourceFlavor`s which Kueue assigned to an admitted `Workload` fails, so the node selector and the tolerations of the flavors cannot be applied. Transport-level errors are skipped and retried every 5 seconds, while a persistent failure is retried with the default interval. |
8081

8182
For a resource held by [`spec.suspend`](spark_custom_resources.md#suspend) or queued by
8283
[Kueue](spark_custom_resources.md#kueue), the operator also publishes the following `Normal`

‎docs/operations.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,10 +47,11 @@ under the License.
4747
`spark.kubernetes.operator.kueue.workloadInformer.enabled`, so that the operator watches
4848
`Workload` resources and an admission starts the queued resource right away. Without the watch,
4949
an admission is noticed on the next periodic reconcile. The watch requires Kueue to be installed
50-
before the operator starts; otherwise the operator reports itself unhealthy. The priority of a
51-
`Workload` comes from the cluster-scoped `workloadpriorityclasses` and `priorityclasses`, which
52-
are granted only by `clusterRole.create`. Without the permission, the operator logs a warning and
53-
creates the `Workload` without priority. See
50+
before the operator starts; otherwise the operator reports itself unhealthy. The cluster-scoped
51+
`resourceflavors`, `workloadpriorityclasses` and `priorityclasses` are granted only by
52+
`clusterRole.create`. Without the permission, the priority of a `Workload` is skipped with a
53+
warning, while the `ResourceFlavor`s which Kueue assigned cannot be read at all, so an admitted
54+
resource is held until the permission is granted. See
5455
[Kueue](spark_custom_resources.md#kueue) for how resources opt into queueing.
5556

5657
## Spark Application Namespaces

‎docs/spark_custom_resources.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -624,6 +624,18 @@ spec:
624624
memory request or limit on the `worker` container, or `SPARK_WORKER_CORES` and
625625
`SPARK_WORKER_MEMORY`: a worker with none of them advertises the whole node to its executors,
626626
well above the default the `Workload` requests.
627+
* Like Kueue built-in integrations, the `nodeLabels` and `tolerations` of the `ResourceFlavor`s
628+
assigned to a pod set are added to the node selector and tolerations of its pods. The executor
629+
pods get them through the executor pod template, which the operator creates if not set. A node
630+
label that conflicts with the node selector of the pods fails the resource with
631+
`SchedulingFailure`, and deletes the `Workload` unless its driver or master is running already,
632+
whose pods still hold the quota it reserved. Only fixing the node selector or the flavor
633+
resolves it: a restarted attempt requests the quota again and hits the same conflict, so an
634+
application which restarts on `SchedulingFailure` should bound the attempts with
635+
[`maxRestartOnSchedulingFailure`](#granular-restart-control). `ResourceFlavor`s are
636+
cluster-scoped, so reading them needs the rules which `operatorRbac.kueue.enabled` grants
637+
through the ClusterRole, hence `operatorRbac.clusterRole.create` as well. Until they can be
638+
read, the resource is held and the read is retried.
627639
* `spec.suspend` takes precedence. A suspended resource does not get a `Workload`, and suspending
628640
a queued resource deletes its `Workload` to release the quota. The resource is queued again when
629641
it is resumed.

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/context/BaseContext.java‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,14 @@
1919

2020
package org.apache.spark.k8s.operator.context;
2121

22+
import java.util.Map;
23+
2224
import io.fabric8.kubernetes.client.KubernetesClient;
2325
import io.javaoperatorsdk.operator.api.event.ResourceEventRecorder;
2426
import io.javaoperatorsdk.operator.api.reconciler.Context;
2527

2628
import org.apache.spark.k8s.operator.BaseResource;
29+
import org.apache.spark.k8s.operator.kueue.KueuePodSetFlavor;
2730

2831
/**
2932
* Base class for context objects.
@@ -35,6 +38,9 @@ public abstract class BaseContext<CR extends BaseResource<?, ?, ?, ?, ?>> {
3538
/** The JOSDK context of the reconciliation this context belongs to. */
3639
protected final Context<?> josdkContext;
3740

41+
/** The flavors which Kueue assigned to the pod sets, applied to the secondary resources. */
42+
protected Map<String, KueuePodSetFlavor> kueuePodSetFlavors = Map.of();
43+
3844
/**
3945
* Constructs a context over the given JOSDK reconciliation context.
4046
*
@@ -51,6 +57,26 @@ protected BaseContext(Context<?> josdkContext) {
5157
*/
5258
public abstract CR getResource();
5359

60+
/**
61+
* Sets the flavors which Kueue assigned to the pod sets of the resource, so that its secondary
62+
* resources carry their node selectors and tolerations.
63+
*
64+
* @param kueuePodSetFlavors The KueuePodSetFlavor by the pod set name.
65+
*/
66+
public void setKueuePodSetFlavors(Map<String, KueuePodSetFlavor> kueuePodSetFlavors) {
67+
synchronized (this) {
68+
this.kueuePodSetFlavors = kueuePodSetFlavors;
69+
applyKueuePodSetFlavors();
70+
}
71+
}
72+
73+
/**
74+
* Applies the flavors to the secondary resource spec which was built before them, e.g. to find
75+
* the driver pod or to check whether the master exists. Called while holding the lock of this
76+
* context, so that a reader sees either spec with its flavors.
77+
*/
78+
protected abstract void applyKueuePodSetFlavors();
79+
5480
/**
5581
* Returns the Kubernetes client associated with this context.
5682
*

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/context/SparkAppContext.java‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,12 +179,25 @@ private SparkAppResourceSpec getSecondaryResourceSpec() {
179179
if (secondaryResourceSpec == null) {
180180
secondaryResourceSpec =
181181
SparkAppResourceSpecFactory.buildResourceSpec(
182-
sparkApplication, josdkContext.getClient(), submissionWorker);
182+
sparkApplication, josdkContext.getClient(), submissionWorker, kueuePodSetFlavors);
183183
}
184184
return secondaryResourceSpec;
185185
}
186186
}
187187

188+
/**
189+
* Drops the resource spec built before the flavors, since they reach the driver and the
190+
* executors through the pod templates which the submission builds the resources from.
191+
*/
192+
@Override
193+
protected void applyKueuePodSetFlavors() {
194+
// The caller holds this lock already, which is re-entrant, so that the field is guarded by
195+
// the same monitor as every other access to it.
196+
synchronized (this) {
197+
secondaryResourceSpec = null;
198+
}
199+
}
200+
188201
/**
189202
* Returns the SparkApplication resource associated with this context.
190203
*

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/context/SparkClusterContext.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@
3131
import org.apache.spark.k8s.operator.SparkCluster;
3232
import org.apache.spark.k8s.operator.SparkClusterResourceSpec;
3333
import org.apache.spark.k8s.operator.SparkClusterSubmissionWorker;
34+
import org.apache.spark.k8s.operator.kueue.KueuePodSetFlavor;
35+
import org.apache.spark.k8s.operator.kueue.KueueWorkloadFactory;
3436
import org.apache.spark.k8s.operator.reconciler.SparkClusterResourceSpecFactory;
3537

3638
/**
@@ -65,11 +67,40 @@ private SparkClusterResourceSpec getSecondaryResourceSpec() {
6567
if (secondaryResourceSpec == null) {
6668
secondaryResourceSpec =
6769
SparkClusterResourceSpecFactory.buildResourceSpec(sparkCluster, submissionWorker);
70+
applyKueuePodSetFlavors();
6871
}
6972
return secondaryResourceSpec;
7073
}
7174
}
7275

76+
/**
77+
* Adds the flavors to the StatefulSets of the resource spec, which is not built again for them:
78+
* unlike the driver and the executors, the master and worker pods are created from the pod
79+
* templates of these StatefulSets. Applying the same flavors twice is a no-op, so the spec built
80+
* with them is left as it is.
81+
*/
82+
@Override
83+
protected void applyKueuePodSetFlavors() {
84+
// The caller holds this lock already, which is re-entrant, so that the field is guarded by
85+
// the same monitor as every other access to it.
86+
synchronized (this) {
87+
if (secondaryResourceSpec == null) {
88+
return;
89+
}
90+
applyKueuePodSetFlavor(
91+
KueueWorkloadFactory.PODSET_MASTER, secondaryResourceSpec.getMasterStatefulSet());
92+
applyKueuePodSetFlavor(
93+
KueueWorkloadFactory.PODSET_WORKER, secondaryResourceSpec.getWorkerStatefulSet());
94+
}
95+
}
96+
97+
private void applyKueuePodSetFlavor(String podSetName, StatefulSet statefulSet) {
98+
KueuePodSetFlavor flavor = kueuePodSetFlavors.get(podSetName);
99+
if (flavor != null) {
100+
flavor.applyTo(statefulSet.getSpec().getTemplate().getSpec());
101+
}
102+
}
103+
73104
/**
74105
* Returns the SparkCluster resource associated with this context.
75106
*

0 commit comments

Comments
 (0)