Skip to content

Commit bfc510c

Browse files
committed
[SPARK-59870] Clarify the Failed message of SparkCluster and the K8s events docs
### What changes were proposed in this pull request? This PR aims to clarify the following messages. - The `Failed` state after `SchedulingFailure` of `SparkCluster` points to `SchedulingFailure` for the cause, instead of `Cannot process cluster status.`. - The description of `spark.kubernetes.operator.events.enabled` refers to `configuration.md` instead of listing only some of the events. - `configuration.md` documents the RBAC of K8s events: `get`, `create` and `patch` on `events`. ### Why are the changes needed? - `Cannot process cluster status.` reads as another, unexplained failure. - The description misses 5 of the 13 non-state event reasons. - Users who manage the operator RBAC without the Helm chart need to know these permissions. ### Does this PR introduce _any_ user-facing change? Yes. The message of the `Failed` state after `SchedulingFailure` of `SparkCluster` changes from `Cannot process cluster status.` in [1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0) (2026-07-23). The rest documents K8s events, which are not released yet. ### How was this patch tested? Pass the CIs with the updated test case. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 Closes #908 from dongjoon-hyun/SPARK-59870. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent cb141d8 commit bfc510c

6 files changed

Lines changed: 42 additions & 11 deletions

File tree

‎docs/config_properties.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
| spark.kubernetes.operator.dynamicConfig.reloadIntervalSeconds | Long | 60 | false | Interval (in seconds) at which the dynamic config file is re-read. Used by the 'file' source. |
1717
| spark.kubernetes.operator.dynamicConfig.selector | String | app.kubernetes.io/name=spark-kubernetes-operator,app.kubernetes.io/component=operator-dynamic-config-overrides | false | The selector str applied to dynamic config map. Used by the 'configMap' source. |
1818
| spark.kubernetes.operator.dynamicConfig.source | String | configMap | false | Source of dynamic config overrides when spark.kubernetes.operator.dynamicConfig.enabled is true. Supported values: 'configMap' (default) watches a ConfigMap via a Kubernetes informer and requires RBAC to read ConfigMaps; 'file' periodically reloads a properties file mounted from a ConfigMap and requires no extra RBAC. |
19-
| spark.kubernetes.operator.events.enabled | Boolean | false | true | When enabled, operator would publish Kubernetes Event objects into the namespace of the Spark resource they describe, making them visible to users of that namespace via 'kubectl describe' and 'kubectl get events'. Events are published when a resource transitions into a new state, as Warning for failure states and Normal for most other states, when the operator cannot reconcile or update the status of a resource, when a resource is held by spec.suspend, when a resource queued by Kueue waits for, gets or fails to request the admission, when the Kueue queue name label of a resource is ignored, and when a Kueue eviction of a running resource is not acted on. These are Event resources in the core API group, unrelated to the internal events that trigger reconciliation. |
19+
| spark.kubernetes.operator.events.enabled | Boolean | false | true | When enabled, operator would publish Kubernetes Event objects into the namespace of the Spark resource they describe, making them visible to users of that namespace via 'kubectl describe' and 'kubectl get events'. Events are published when a resource transitions into a new state, as Warning for failure states and Normal for most other states, and on other occasions, e.g. when an operation on a resource fails, or while a resource is held by spec.suspend or Kueue. See the 'Kubernetes Events' section of docs/configuration.md for all reasons and the required RBAC permissions. These are Event resources in the core API group, unrelated to the internal events that trigger reconciliation. |
2020
| spark.kubernetes.operator.events.excludedReasons | String | | true | Comma-separated list of Java regular expressions of Kubernetes Event reasons that operator would not publish even when 'spark.kubernetes.operator.events.enabled' is true, e.g. 'RunningHealthy,RunningWithPartialCapacity' or 'Running.*'. Each expression must match the whole reason and is case-sensitive. An invalid expression matches only the reason identical to it. If empty, operator would publish events of all reasons. |
2121
| spark.kubernetes.operator.events.minIntervalSeconds | Long | 300 | true | Minimum interval (in seconds) between two Kubernetes Events with the same reason and the same message on the same Spark resource. The first event of a reason on a resource is always published right away, only a repeat that says exactly what the last one said is dropped. Such a repeat only bumps the 'count' of the one Event object, so it carries no information while it still costs a read and a write on the API server. A repeat whose message differs, for instance because it embeds the cause of a failure, is always published, since the operator rewrites the message of the existing Event. An event is only published when a reconciliation emits it, so the effective period is this interval rounded up to the next repeat. Keep this interval plus the interval at which the repeats are emitted within the '--event-ttl' of the API server (one hour by default), or a repeat lands after the previous event expired and the hold stops being visible. The repeats are emitted every 'spark.kubernetes.operator.reconciler.suspendHoldRequeueIntervalSeconds' for SuspendHeld and every 'spark.kubernetes.operator.reconciler.intervalSeconds' for KueueAdmissionPending, so with the defaults this option may be at most 1800 and 3480 seconds respectively. Unlike 'spark.kubernetes.operator.events.excludedReasons', which blocks a reason entirely, this only limits how often it is published. If zero or negative, operator would publish every event. |
2222
| spark.kubernetes.operator.health.probePort | Integer | 19091 | false | The port used for health/readiness check probe status. |

‎docs/configuration.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,19 @@ When `spark.kubernetes.operator.events.enabled` is `true`, the operator publishe
5757
`Event` objects about `SparkApplication` and `SparkCluster` resources into their namespaces. They
5858
are visible via `kubectl describe` and `kubectl get events`.
5959

60+
Publishing them requires the `get`, `create` and `patch` permissions on `events` in the core API
61+
group in each namespace of the Spark resources. Without them, the operator logs a
62+
`Could not record ... event` warning for each event that it cannot publish, and keeps reconciling.
63+
The operator ClusterRole and Roles created by the Helm chart include them. If you manage the RBAC
64+
of the operator yourself, e.g. with a custom `operatorRbac.roleBinding.roleRef`, add the following
65+
rule to its ClusterRole, or to its Role in each of those namespaces.
66+
67+
```yaml
68+
- apiGroups: [""]
69+
resources: ["events"]
70+
verbs: ["get", "create", "patch"]
71+
```
72+
6073
Every event is reported by `spark-kubernetes-operator`, which is set as both `source.component` and
6174
`reportingComponent`, so `kubectl describe` shows it in the `From` column and both the `source` and
6275
the `reportingComponent` field selectors match it. For example, the following lists the warnings

‎spark-operator-api/src/main/java/org/apache/spark/k8s/operator/Constants.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,6 +273,14 @@ public final class Constants {
273273
public static final String CLUSTER_REQUEUED_MESSAGE =
274274
"Cluster is submitted again after its eviction by Kueue.";
275275

276+
/**
277+
* Message indicating that the cluster failed after SchedulingFailure, whose message tells the
278+
* cause.
279+
*/
280+
public static final String CLUSTER_FAILED_AFTER_SCHEDULING_FAILURE_MESSAGE =
281+
"Cluster failed, since requesting its resources failed. See the preceding "
282+
+ "SchedulingFailure state for the cause.";
283+
276284
/** Message indicating that the cluster status cannot be processed. */
277285
public static final String UNKNOWN_CLUSTER_STATE_MESSAGE = "Cannot process cluster status.";
278286

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/config/SparkOperatorConf.java‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -332,12 +332,11 @@ public final class SparkOperatorConf {
332332
+ "of the Spark resource they describe, making them visible to users of that "
333333
+ "namespace via 'kubectl describe' and 'kubectl get events'. Events are "
334334
+ "published when a resource transitions into a new state, as Warning for "
335-
+ "failure states and Normal for most other states, when the operator "
336-
+ "cannot reconcile or update the status of a resource, when a resource is "
337-
+ "held by spec.suspend, when a resource queued by Kueue waits for, gets or "
338-
+ "fails to request the admission, when the Kueue queue name label of a "
339-
+ "resource is ignored, and when a Kueue eviction of a running resource is not "
340-
+ "acted on. These are Event resources in the core "
335+
+ "failure states and Normal for most other states, and on other occasions, "
336+
+ "e.g. when an operation on a resource fails, or while a resource is held by "
337+
+ "spec.suspend or Kueue. See the 'Kubernetes Events' section of "
338+
+ "docs/configuration.md for all reasons and the required RBAC permissions. "
339+
+ "These are Event resources in the core "
341340
+ "API group, unrelated to the internal events that trigger reconciliation.")
342341
.typeParameterClass(Boolean.class)
343342
.defaultValue(false)

‎spark-operator/src/main/java/org/apache/spark/k8s/operator/reconciler/reconcilesteps/ClusterUnknownStateStep.java‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,19 @@ public final class ClusterUnknownStateStep extends ClusterReconcileStep {
3939
@Override
4040
public ReconcileProgress reconcile(
4141
SparkClusterContext context, SparkClusterStatusRecorder statusRecorder) {
42+
ClusterStateSummary currentStateSummary =
43+
context.getResource().getStatus().getCurrentState().getCurrentStateSummary();
4244
// Appending Failed again would write a new status on every reconcile, forever.
43-
if (context.getResource().getStatus().getCurrentState().getCurrentStateSummary()
44-
== ClusterStateSummary.Failed) {
45+
if (currentStateSummary == ClusterStateSummary.Failed) {
4546
return ReconcileProgress.completeAndNoRequeue();
4647
}
47-
ClusterState state =
48-
new ClusterState(ClusterStateSummary.Failed, Constants.UNKNOWN_CLUSTER_STATE_MESSAGE);
48+
// SchedulingFailure already reports the cause, so Failed points back to it rather than reading
49+
// as another, unexplained failure.
50+
String message =
51+
currentStateSummary == ClusterStateSummary.SchedulingFailure
52+
? Constants.CLUSTER_FAILED_AFTER_SCHEDULING_FAILURE_MESSAGE
53+
: Constants.UNKNOWN_CLUSTER_STATE_MESSAGE;
54+
ClusterState state = new ClusterState(ClusterStateSummary.Failed, message);
4955
statusRecorder.persistStatus(context, context.getResource().getStatus().appendNewState(state));
5056
return ReconcileProgress.completeAndImmediateRequeue();
5157
}

‎spark-operator/src/test/java/org/apache/spark/k8s/operator/reconciler/SparkClusterReconcilerTest.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
package org.apache.spark.k8s.operator.reconciler;
2121

22+
import static org.apache.spark.k8s.operator.Constants.CLUSTER_FAILED_AFTER_SCHEDULING_FAILURE_MESSAGE;
2223
import static org.apache.spark.k8s.operator.Constants.LABEL_SPARK_CLUSTER_NAME;
2324
import static org.apache.spark.k8s.operator.utils.TestUtils.setConfigKey;
2425
import static org.assertj.core.api.Assertions.assertThat;
@@ -261,6 +262,10 @@ void testReconcileFailedClusterDoesNotLoop() throws Exception {
261262
assertEquals(
262263
ClusterStateSummary.Failed,
263264
cluster.getStatus().getCurrentState().getCurrentStateSummary());
265+
// Failed points back to SchedulingFailure, which reports the cause.
266+
assertEquals(
267+
CLUSTER_FAILED_AFTER_SCHEDULING_FAILURE_MESSAGE,
268+
cluster.getStatus().getCurrentState().getMessage());
264269
int historySize = cluster.getStatus().getStateTransitionHistory().size();
265270
verify(mockRecorder, times(1)).persistStatus(any(), any());
266271

0 commit comments

Comments
 (0)