Skip to content

Commit 42cf2ec

Browse files
committed
[SPARK-59677] Support K8s events for suspended SparkApplication and SparkCluster
### What changes were proposed in this pull request? This PR aims to support K8s events for `SparkApplication` and `SparkCluster` held by `spec.suspend`. | Type | Reason | When | |---|---|---| | `Normal` | `SuspendHeld` | The resource is held by `spec.suspend`, so the driver (or master and worker) is not requested | Like the `KueueAdmissionPending` event of SPARK-59638, it is republished while the hold lasts. The event sink keys the `Event` on the reason, so a repeat bumps the `count` of the one `Event` and refreshes it instead of creating another. An application held in `ScheduledToRestart` gets the same event, because the status of its previous attempt says that a restart is due, not that the next attempt is withheld. Resuming publishes no event of its own, since the following state transition already reports it. Unlike the Kueue hold, which ends when quota arrives, this one ends only when a user clears `spec.suspend`. Republishing on every reconcile would therefore cost a read and a write on the API server for as long as a resource is parked, so the hold is requeued every 30 minutes instead, which still refreshes the event well within the one hour that the API server retains events by default. Nothing waits for that interval, since clearing `spec.suspend` arrives as a watch event. The reason is `SuspendHeld` rather than `Suspended` because state transition events use the state name as both the reason and the dedup key, so the latter would collide with the dedicated `Suspended` state planned in a follow-up of SPARK-59542. In addition, - `SuspendUtils.holdForSuspend(BaseContext<?>, String)` holds the policy for both init steps, mirroring how SPARK-59638 extracted `KueueWorkloadUtils.holdForAdmission`. Each step keeps only its own guard. - Suspending a queued resource releases its Kueue `Workload`, and the message now says so: the `KueueAdmissionPending` event it was queued with has no owner reference and outlives that `Workload`, so it would otherwise contradict the new event until the API server drops it. - A new `suspend-events` E2E group asserts the event for both resource kinds, and a new `events` CI mode installs the operator with `spark.kubernetes.operator.events.enabled=true` through `tests/e2e/helm/events-config-values.yaml`. The existing `suspend` group keeps running with the shipped defaults. - The description of `spark.kubernetes.operator.events.enabled` covers the new event. ### Why are the changes needed? A suspended resource has no state transition, so users cannot see why it is stuck via `kubectl describe`. The initial `Submitted` status of a resource created with `suspend: true` is not persisted either, so `kubectl get` shows an empty `Current State` as well. Publishing the event once would not be enough: Kubernetes retains events only for a limited time (one hour by default), so a resource suspended for longer would end up with neither a status nor an event. ### Does this PR introduce _any_ user-facing change? Yes. When `spark.kubernetes.operator.events.enabled` is `true`, the new event is published for resources held by `spec.suspend`. To opt out, add `SuspendHeld` to `spark.kubernetes.operator.events.excludedReasons`. ### How was this patch tested? Pass the CIs with the newly added unit tests in `AppInitStepTest` and `ClusterInitStepTest`, and the new `suspend-events` E2E group which asserts the event for both `SparkApplication` and `SparkCluster`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 Closes #854 from dongjoon-hyun/SPARK-59677. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 4147887 commit 42cf2ec

17 files changed

Lines changed: 686 additions & 56 deletions

File tree

‎.github/workflows/build_and_test.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,9 @@ jobs:
145145
- kubernetes-version: "1.37.0"
146146
mode: static
147147
test-group: pi-java25
148+
- kubernetes-version: "1.37.0"
149+
mode: events
150+
test-group: suspend-events
148151
- kubernetes-version: "1.37.0"
149152
mode: kueue
150153
test-group: kueue
@@ -244,6 +247,21 @@ jobs:
244247
if: matrix.mode == 'selector'
245248
run: |
246249
chainsaw test --test-dir ./tests/e2e/${{ matrix.test-group }} --parallel 1
250+
- name: Run Spark K8s Operator on K8S with Kubernetes Events Enabled
251+
if: matrix.mode == 'events'
252+
run: |
253+
eval $(minikube docker-env)
254+
./gradlew buildDockerImage
255+
helm install spark --create-namespace -f \
256+
build-tools/helm/spark-kubernetes-operator/values.yaml -f \
257+
tests/e2e/helm/events-config-values.yaml \
258+
build-tools/helm/spark-kubernetes-operator/
259+
helm test spark
260+
minikube docker-env --unset
261+
- name: Run E2E Test with Kubernetes Events Enabled
262+
if: matrix.mode == 'events'
263+
run: |
264+
chainsaw test --test-dir ./tests/e2e/${{ matrix.test-group }} --parallel 1
247265
- name: Run Spark K8s Operator on K8S with Kueue Enabled
248266
if: matrix.mode == 'kueue'
249267
run: |

‎docs/config_properties.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
| spark.kubernetes.operator.dynamicConfig.reloadIntervalSeconds | Long | 60 | false | Interval (in seconds) at which the dynamic config file is re-read. Used by the 'file' source. |
1717
| spark.kubernetes.operator.dynamicConfig.selector | String | app.kubernetes.io/name=spark-kubernetes-operator,app.kubernetes.io/component=operator-dynamic-config-overrides | false | The selector str applied to dynamic config map. Used by the 'configMap' source. |
1818
| spark.kubernetes.operator.dynamicConfig.source | String | configMap | false | Source of dynamic config overrides when spark.kubernetes.operator.dynamicConfig.enabled is true. Supported values: 'configMap' (default) watches a ConfigMap via a Kubernetes informer and requires RBAC to read ConfigMaps; 'file' periodically reloads a properties file mounted from a ConfigMap and requires no extra RBAC. |
19-
| spark.kubernetes.operator.events.enabled | Boolean | false | true | When enabled, operator would publish Kubernetes Event objects into the namespace of the Spark resource they describe, making them visible to users of that namespace via 'kubectl describe' and 'kubectl get events'. Events are published when a resource transitions into a new state, as Warning for failure states and Normal for most other states, when the operator cannot reconcile or update the status of a resource, and when a resource queued by Kueue waits for, gets or fails to request the admission. These are Event resources in the core API group, unrelated to the internal events that trigger reconciliation. |
19+
| spark.kubernetes.operator.events.enabled | Boolean | false | true | When enabled, operator would publish Kubernetes Event objects into the namespace of the Spark resource they describe, making them visible to users of that namespace via 'kubectl describe' and 'kubectl get events'. Events are published when a resource transitions into a new state, as Warning for failure states and Normal for most other states, when the operator cannot reconcile or update the status of a resource, when a resource is held by spec.suspend, and when a resource queued by Kueue waits for, gets or fails to request the admission. These are Event resources in the core API group, unrelated to the internal events that trigger reconciliation. |
2020
| spark.kubernetes.operator.events.excludedReasons | String | | true | Comma-separated list of Java regular expressions of Kubernetes Event reasons that operator would not publish even when 'spark.kubernetes.operator.events.enabled' is true, e.g. 'RunningHealthy,RunningWithPartialCapacity' or 'Running.*'. Each expression must match the whole reason and is case-sensitive. An invalid expression matches only the reason identical to it. If empty, operator would publish events of all reasons. |
2121
| spark.kubernetes.operator.health.probePort | Integer | 19091 | false | The port used for health/readiness check probe status. |
2222
| spark.kubernetes.operator.health.sentinelExecutorPoolSize | Integer | 3 | false | Size of executor service in Sentinel Managers to check the health of sentinel resources. |
@@ -52,6 +52,7 @@
5252
| spark.kubernetes.operator.reconciler.retry.initialIntervalSeconds | Integer | 5 | false | Initial interval(in seconds) of retries on unhandled controller errors. |
5353
| spark.kubernetes.operator.reconciler.retry.intervalMultiplier | Double | 1.5 | false | Interval multiplier of retries on unhandled controller errors. Setting this to 1 for linear retry. |
5454
| spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds | Integer | -1 | false | Max interval(in seconds) of retries on unhandled controller errors. Set to non-positive for unlimited. |
55+
| spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds | Long | 1800 | true | Requeue interval (in seconds) at which a resource held by spec.suspend is reconciled, so that its SuspendHeld event is republished. The hold ends only when a user clears spec.suspend, which arrives as a watch event and reconciles right away, so nothing waits for this interval. It is deliberately much coarser than spark.kubernetes.operator.reconciler.intervalSeconds, since a suspended resource has nothing to observe while each republish costs a read and a write on the API server. Keep it below the '--event-ttl' of the API server (one hour by default), or the event expires between repeats and the hold stops being visible. |
5556
| spark.kubernetes.operator.reconciler.terminationTimeoutSeconds | Integer | 30 | false | Grace period for operator shutdown before reconciliation threads are killed. |
5657
| spark.kubernetes.operator.reconciler.trimStateTransitionHistoryEnabled | Boolean | true | true | When enabled, operator would trim state transition history when a new attempt starts, keeping previous attempt summary only. |
5758
| spark.kubernetes.operator.watchedNamespaces | String | default | true | Comma-separated list of namespaces that the operator would be watching for Spark resources. If set to '*', operator would watch all namespaces. |

‎docs/configuration.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,17 +76,20 @@ In addition, the operator publishes the following `Warning` events.
7676
| `StatusUpdateFailed` | A status patch is rejected. Transport-level errors are skipped. |
7777
| `KueueAdmissionRequestFailed` | Creating, reading or deleting a stale Kueue `Workload` fails. Transport-level errors are skipped and retried every 5 seconds, while a persistent failure is retried with the default interval. |
7878

79-
For a resource queued by [Kueue](spark_custom_resources.md#kueue), the operator also publishes the
80-
following `Normal` events, since the resource stays in its initializing state while it waits.
79+
For a resource held by [`spec.suspend`](spark_custom_resources.md#suspend) or queued by
80+
[Kueue](spark_custom_resources.md#kueue), the operator also publishes the following `Normal`
81+
events, since the resource stays in its initializing state without a state transition.
8182

8283
| Reason | When |
8384
|---|---|
85+
| `SuspendHeld` | The resource is held by `spec.suspend`, so the driver (or master and worker) is not requested. It is republished every 30 minutes while the hold lasts (`spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds`), so a repeat bumps the `count` of the one event rather than creating another. Suspending a queued resource releases its Kueue `Workload`, and the message says so, since the `KueueAdmissionPending` event it was queued with outlives that `Workload`. |
8486
| `KueueAdmissionPending` | The Kueue `Workload` waits for the admission. It is republished while it waits, so a repeat bumps the `count` of the one event rather than creating another. |
8587
| `KueueAdmitted` | Kueue admitted the `Workload`, so the driver (or master and worker) is requested. |
8688

8789
The `reason` values are stable, while the `message` values may change between releases. Note that
8890
Kubernetes retains events only for a limited time (one hour by default), so the resource status
89-
remains the source of truth.
91+
remains the source of truth. The exception is a resource held by `spec.suspend` or by Kueue, whose
92+
status does not report the hold; that is why its event is republished while the hold lasts.
9093

9194
To reduce noise, set `spark.kubernetes.operator.events.excludedReasons` to a comma-separated list
9295
of Java regular expressions. The operator does not publish events whose `reason` fully matches any

‎docs/spark_custom_resources.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -536,8 +536,13 @@ StatefulSets. Setting it back to `false` resumes the regular lifecycle.
536536

537537
`Submitted` here is the operator's in-memory view. For a valid resource created with
538538
`suspend: true`, the initial `Submitted` status is not persisted to the API server, so
539-
`kubectl get` shows an empty `Current State` until initialization resumes. An application held
540-
later, in `ScheduledToRestart`, keeps the status its previous attempt already wrote.
539+
`kubectl get` shows an empty `Current State` and no state transition events are published until
540+
initialization resumes. Instead, the `SuspendHeld` [event](configuration.md#kubernetes-events) is
541+
published when enabled. Since the status is not there to fall back on, it is republished every 30
542+
minutes by default while the hold lasts, so that it outlives the event retention of the API
543+
server. An application held later, in `ScheduledToRestart`, keeps the status its previous attempt
544+
already wrote and gets the same event, since that status says that a restart is due, not that the
545+
next attempt is withheld.
541546

542547
``` yaml
543548
apiVersion: spark.apache.org/v1

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java‎

Lines changed: 33 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,35 @@ public final class SparkOperatorConf {
262262
.defaultValue(15)
263263
.build();
264264

265+
/**
266+
* Requeue interval (in seconds) at which a resource held by {@code spec.suspend} is reconciled,
267+
* so that its {@code SuspendHeld} event is republished. The hold ends only when a user clears
268+
* {@code spec.suspend}, which arrives as a watch event and reconciles right away, so nothing
269+
* waits for this interval. It is deliberately much coarser than {@link
270+
* #RECONCILER_INTERVAL_SECONDS}, since a suspended resource has nothing to observe while each
271+
* republish costs a read and a write on the API server. Keep it below the {@code --event-ttl} of
272+
* the API server (one hour by default), or the event expires between repeats and the hold stops
273+
* being visible.
274+
*/
275+
public static final ConfigOption<Long> SUSPEND_HOLD_REQUEUE_INTERVAL_SECONDS =
276+
ConfigOption.<Long>builder()
277+
.key("spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds")
278+
.enableDynamicOverride(true)
279+
.description(
280+
"Requeue interval (in seconds) at which a resource held by spec.suspend is "
281+
+ "reconciled, so that its SuspendHeld event is republished. The hold ends only "
282+
+ "when a user clears spec.suspend, which arrives as a watch event and "
283+
+ "reconciles right away, so nothing waits for this interval. It is "
284+
+ "deliberately much coarser than "
285+
+ "spark.kubernetes.operator.reconciler.intervalSeconds, since a suspended "
286+
+ "resource has nothing to observe while each republish costs a read and a "
287+
+ "write on the API server. Keep it below the '--event-ttl' of the API server "
288+
+ "(one hour by default), or the event expires between repeats and the hold "
289+
+ "stops being visible.")
290+
.typeParameterClass(Long.class)
291+
.defaultValue(1800L)
292+
.build();
293+
265294
/**
266295
* When enabled, operator would trim state transition history when a new attempt starts, keeping
267296
* previous attempt summary only.
@@ -292,10 +321,10 @@ public final class SparkOperatorConf {
292321
+ "namespace via 'kubectl describe' and 'kubectl get events'. Events are "
293322
+ "published when a resource transitions into a new state, as Warning for "
294323
+ "failure states and Normal for most other states, when the operator "
295-
+ "cannot reconcile or update the status of a resource, and when a resource "
296-
+ "queued by Kueue waits for, gets or fails to request the admission. These "
297-
+ "are Event resources in the core API group, unrelated to the internal "
298-
+ "events that trigger reconciliation.")
324+
+ "cannot reconcile or update the status of a resource, when a resource is "
325+
+ "held by spec.suspend, and when a resource queued by Kueue waits for, gets "
326+
+ "or fails to request the admission. These are Event resources in the core "
327+
+ "API group, unrelated to the internal events that trigger reconciliation.")
299328
.typeParameterClass(Boolean.class)
300329
.defaultValue(false)
301330
.build();

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/context/SparkAppContext.java‎

Lines changed: 31 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -97,11 +97,32 @@ public Optional<Pod> getDriverPod() {
9797
* <p>The informer cache is only used to find a candidate. Since the cache may still hold the
9898
* pre-deletion snapshot of a previous attempt's pod, the candidate is verified against the API
9999
* server and returned only if it exists there and is not terminating. If the verification fails
100-
* because of an API error, the pod is considered absent for this reconciliation.
100+
* because of an API error, the pod is considered absent for this reconciliation. Callers that
101+
* must not act on that assumption use {@link #getCurrentAttemptDriverPodStrictly()} instead.
101102
*
102103
* @return An Optional containing the driver Pod of the current attempt, or empty if not found.
103104
*/
104105
public Optional<Pod> getCurrentAttemptDriverPod() {
106+
try {
107+
return getCurrentAttemptDriverPodStrictly();
108+
} catch (KubernetesClientException e) {
109+
log.warn(
110+
"Failed to verify driver pod {} against the API server, considering it absent.",
111+
getDriverPodSpec().getMetadata().getName(),
112+
e);
113+
return Optional.empty();
114+
}
115+
}
116+
117+
/**
118+
* Same as {@link #getCurrentAttemptDriverPod()}, but reports a failed verification instead of
119+
* reading it as an absent pod. A caller that would otherwise conclude that no driver has been
120+
* requested, and act on that, uses this so that an API error does not look like an answer.
121+
*
122+
* @return An Optional containing the driver Pod of the current attempt, or empty if not found.
123+
* @throws KubernetesClientException if the candidate cannot be verified against the API server.
124+
*/
125+
public Optional<Pod> getCurrentAttemptDriverPodStrictly() {
105126
List<Pod> driverPods =
106127
josdkContext
107128
.getSecondaryResourcesAsStream(Pod.class)
@@ -120,25 +141,17 @@ public Optional<Pod> getCurrentAttemptDriverPod() {
120141
if (driverPods.stream().noneMatch(p -> driverPodName.equals(p.getMetadata().getName()))) {
121142
return Optional.empty();
122143
}
123-
try {
124-
Pod livePod =
125-
josdkContext
126-
.getClient()
127-
.pods()
128-
.inNamespace(sparkApplication.getMetadata().getNamespace())
129-
.withName(driverPodName)
130-
.get();
131-
if (livePod == null || livePod.getMetadata().getDeletionTimestamp() != null) {
132-
return Optional.empty();
133-
}
134-
return Optional.of(livePod);
135-
} catch (KubernetesClientException e) {
136-
log.warn(
137-
"Failed to verify driver pod {} against the API server, considering it absent.",
138-
driverPodName,
139-
e);
144+
Pod livePod =
145+
josdkContext
146+
.getClient()
147+
.pods()
148+
.inNamespace(sparkApplication.getMetadata().getNamespace())
149+
.withName(driverPodName)
150+
.get();
151+
if (livePod == null || livePod.getMetadata().getDeletionTimestamp() != null) {
140152
return Optional.empty();
141153
}
154+
return Optional.of(livePod);
142155
}
143156

144157
/**

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/kueue/KueueWorkloadUtils.java‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -357,22 +357,26 @@ public static boolean isAdmissionChanged(final Workload newWorkload, final Workl
357357
}
358358

359359
/**
360-
* Deletes the Workload of the given resource so that Kueue releases its quota. Failures are
361-
* logged only, because the Workload is garbage collected with its owner anyway.
360+
* Deletes the Workload of the given resource so that Kueue releases its quota, and reports
361+
* whether one was there to delete. Failures are logged only, because the Workload is garbage
362+
* collected with its owner anyway.
362363
*
363364
* @param client The KubernetesClient.
364365
* @param owner The SparkApplication or SparkCluster owning the Workload.
366+
* @return True if a Workload was deleted, false if there was none or the deletion failed.
365367
*/
366-
public static void releaseWorkload(final KubernetesClient client, final HasMetadata owner) {
368+
public static boolean releaseWorkload(final KubernetesClient client, final HasMetadata owner) {
367369
String name = KueueWorkloadFactory.getWorkloadName(owner);
368370
try {
369-
client
371+
return !client
370372
.resources(Workload.class)
371373
.inNamespace(owner.getMetadata().getNamespace())
372374
.withName(name)
373-
.delete();
375+
.delete()
376+
.isEmpty();
374377
} catch (KubernetesClientException e) {
375378
log.warn("Failed to release the Kueue Workload {}.", name, e);
379+
return false;
376380
}
377381
}
378382

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/AppInitStep.java‎

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434

3535
import io.fabric8.kubernetes.api.model.HasMetadata;
3636
import io.fabric8.kubernetes.api.model.Pod;
37+
import io.fabric8.kubernetes.client.KubernetesClientException;
3738
import lombok.extern.slf4j.Slf4j;
3839

3940
import org.apache.spark.k8s.operator.Constants;
@@ -69,13 +70,20 @@ public ReconcileProgress reconcile(
6970
return proceed();
7071
}
7172
SparkApplication app = context.getResource();
72-
if (app.getSpec().isSuspend() && !isDriverRequested(context)) {
73-
log.debug("Application is suspended, driver resources would not be requested.");
74-
if (KueueWorkloadFactory.hasQueueName(app)) {
75-
// A resource suspended while queued must not keep holding the Kueue quota.
76-
KueueWorkloadUtils.releaseWorkload(context.getClient(), app);
73+
if (app.getSpec().isSuspend()) {
74+
Optional<Pod> currentAttemptDriverPod;
75+
try {
76+
currentAttemptDriverPod = context.getCurrentAttemptDriverPodStrictly();
77+
} catch (KubernetesClientException e) {
78+
// Whether a driver of this attempt is live is unknown, not answered. Holding would claim
79+
// in an event that none was requested, and would do so for the whole suspend hold
80+
// interval, so look again with the steady-state interval instead.
81+
log.warn("Failed to verify the driver pod of a suspended application, will retry.", e);
82+
return completeAndDefaultRequeue();
83+
}
84+
if (currentAttemptDriverPod.isEmpty()) {
85+
return SuspendUtils.holdForSuspend(context, "driver");
7786
}
78-
return completeAndDefaultRequeue();
7987
}
8088
if (app.getStatus().getPreviousAttemptSummary() != null) {
8189
Instant lastTransitionTime = Instant.parse(currentState.getLastTransitionTime());

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterInitStep.java‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -66,12 +66,7 @@ public ReconcileProgress reconcile(
6666
// A cluster whose master StatefulSet already exists has been requested before (e.g. the status
6767
// update to RunningHealthy failed), so let it complete its initialization even if suspended.
6868
if (cluster.getSpec().isSuspend() && !isMasterRequested(context)) {
69-
log.debug("Cluster is suspended, master and worker resources would not be requested.");
70-
if (KueueWorkloadFactory.hasQueueName(cluster)) {
71-
// A resource suspended while queued must not keep holding the Kueue quota.
72-
KueueWorkloadUtils.releaseWorkload(context.getClient(), cluster);
73-
}
74-
return completeAndDefaultRequeue();
69+
return SuspendUtils.holdForSuspend(context, "master and workers");
7570
}
7671
if (cluster.getStatus().getPreviousAttemptSummary() != null) {
7772
Instant lastTransitionTime = Instant.parse(currentState.getLastTransitionTime());

0 commit comments

Comments
 (0)