From 1f4fcdbe97f0947873442cee668a6d5412c6e4dc Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Wed, 16 Sep 2026 14:27:31 -0700 Subject: [PATCH 1/2] [SPARK-59596] Fix Helm chart configuration table in `operations.md` to match `values.yaml` --- docs/operations.md | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/docs/operations.md b/docs/operations.md index 11a4d099e..444d59856 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -83,25 +83,32 @@ following table: | image.tag | The image tag of spark-kubernetes-operator. | 1.1.0-SNAPSHOT | | image.digest | The image digest of spark-kubernetes-operator. If set then it takes precedence and the image tag will be ignored. | | | imagePullSecrets | The image pull secrets of spark-kubernetes-operator. | | -| operatorDeployment.replica | Operator replica count. Must be 1 unless leader election is configured. | 1 | +| operatorDeployment.replicas | Operator replica count. Must be 1 unless leader election is configured. | 1 | | operatorDeployment.strategy.type | Operator pod upgrade strategy. Must be Recreate unless leader election is configured. | Recreate | +| operatorDeployment.networkPolicy.enabled | When enabled, a NetworkPolicy allows ingress to the operator pod only on the health probe and metrics ports. Requires a CNI plugin with NetworkPolicy support. | false | +| operatorDeployment.networkPolicy.metricsIngress | List of NetworkPolicyPeer(s) allowed to reach the metrics port, e.g. the Prometheus scraper. When empty, ingress to the metrics port is denied. | | | operatorDeployment.operatorPod.annotations | Custom annotations to be added to the operator pod | | | operatorDeployment.operatorPod.labels | Custom labels to be added to the operator pod | | +| operatorDeployment.operatorPod.affinity | Custom affinity to be added to the operator pod. | | | operatorDeployment.operatorPod.nodeSelector | Custom nodeSelector to be added to the operator pod. | | +| operatorDeployment.operatorPod.tolerations | Custom tolerations to be added to the operator pod. | | | operatorDeployment.operatorPod.topologySpreadConstraints | Custom topologySpreadConstraints to be added to the operator pod. | | +| operatorDeployment.operatorPod.dnsPolicy | DNS policy to be used by the operator pod. | | | operatorDeployment.operatorPod.dnsConfig | DNS configuration to be used by the operator pod. | | | operatorDeployment.operatorPod.volumes | Additional volumes to be added to the operator pod. | | | operatorDeployment.operatorPod.priorityClassName | Priority class name to be used for the operator pod | | | operatorDeployment.operatorPod.securityContext | Security context overrides for the operator pod | | -| operatorDeployment.operatorPod.operatorContainer.jvmArgs | JVM arg override for the operator container. | `"-Dfile.encoding=UTF8 -XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/dev/stderr -XX:+UseParallelGC"` | +| operatorDeployment.operatorPod.operatorContainer.jvmArgs | JVM arg override for the operator container. | `"-Dfile.encoding=UTF8 -XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/dev/stderr -XX:+UseParallelGC -XX:InitialRAMPercentage=80 -XX:MaxRAMPercentage=80 -XX:+AlwaysPreTouch -XX:+UseCompactObjectHeaders"` | | operatorDeployment.operatorPod.operatorContainer.env | Custom env to be added to the operator container. | | | operatorDeployment.operatorPod.operatorContainer.envFrom | Custom envFrom to be added to the operator container, e.g. for downward API. | | +| operatorDeployment.operatorPod.operatorContainer.volumeMounts | Additional volume mounts to be added to the operator container. | | | operatorDeployment.operatorPod.operatorContainer.probes | Probe config for the operator container. | | +| operatorDeployment.operatorPod.operatorContainer.metrics.port | Metrics port of the operator container. | 19090 | | operatorDeployment.operatorPod.operatorContainer.securityContext | Security context overrides for the operator container. | run as non root for baseline security standard compliance | | operatorDeployment.operatorPod.operatorContainer.resources | Resources for the operator container. | memory 2Gi, ephemeral storage 2Gi and 1 cpu | -| operatorDeployment.additionalContainers | Additional containers to be added to the operator pod, e.g. sidecar. | | +| operatorDeployment.operatorPod.additionalContainers | Additional containers to be added to the operator pod, e.g. sidecar. | | | operatorRbac.serviceAccount.create | Whether to create service account for operator to use. | true | -| operatorRbac.serviceAccount.name | Name of the operator Role. | `"spark-operator"` | +| operatorRbac.serviceAccount.name | Name of the operator ServiceAccount. | `"spark-operator"` | | operatorRbac.clusterRole.create | Whether to create ClusterRole for operator to use. | true | | operatorRbac.clusterRole.name | Name of the operator ClusterRole. | `"spark-operator-clusterrole"` | | operatorRbac.clusterRoleBinding.create | Whether to create ClusterRoleBinding for operator to use. | true | @@ -111,11 +118,12 @@ following table: | operatorRbac.roleBinding.create | Whether to create RoleBinding for operator to use. At least one of `clusterRoleBinding.create` or `roleBinding.create` should be enabled | false | | operatorRbac.roleBinding.name | Name of the operator RoleBinding in each workload namespace(s). | `"spark-operator-rolebinding"` | | operatorRbac.roleBinding.roleRef | RoleRef for the created Operator RoleBinding. Override this when you want the created RoleBinding refer to ClusterRole / Role that's different from the default operator Role. | Refers to default `operatorRbac.role.name` | -| operatorRbac.configManagement.create | Enable this to create a Role for operator configuration management (hot property loading and leader election). | true | -| operatorRbac.configManagement.roleName | Role name for operator configuration management. | `spark-operator-config-role` | -| operatorRbac.configManagement.roleBinding | RoleBinding name for operator configuration management. | `"spark-operator-config-monitor-role-binding"` | +| operatorRbac.configManagement.create | Enable this to create a Role for operator configuration management (hot property loading from ConfigMap). Requires `dynamicConfig` with the `configMap` source. | true | +| operatorRbac.configManagement.roleName | Role name for operator configuration management. | `"spark-operator-config-monitor"` | +| operatorRbac.configManagement.roleBindingName | RoleBinding name for operator configuration management. | `"spark-operator-config-monitor-role-binding"` | | operatorRbac.kueue.enabled | Grant the operator access to Kueue `workloads`, `resourceflavors`, `workloadpriorityclasses` and to `priorityclasses`. The cluster-scoped ones need `clusterRole.create`. See [Optional Prerequisites](#optional-prerequisites). | false | | operatorRbac.labels | Labels to be applied on all created `operatorRbac` resources. | `"app.kubernetes.io/component": "operator-rbac"` | +| operatorRbac.annotations | Annotations to be applied on all created `operatorRbac` resources. | | | workloadResources.namespaces.create | Whether to create dedicated namespaces for Spark workload. | true | | workloadResources.namespaces.overrideWatchedNamespaces | When enabled, operator would by default only watch namespace(s) provided in data field. | true | | workloadResources.namespaces.data | List of namespaces to create for Spark workload. The chart namespace would be used if this is empty. | | @@ -125,23 +133,26 @@ following table: | workloadResources.role.name | Name for Spark workload Role. | "spark-workload-role" | | workloadResources.roleBinding.create | When enabled, a RoleBinding would be created in each namespace for Spark workload. This shall be enabled unless access is configured from 3rd party. | true | | workloadResources.roleBinding.name | Name of the Spark workload RoleBinding. | "spark-workload-rolebinding" | -| workloadResources.serviceAccounts.create | Whether to create a service account for Spark workload. | true | -| workloadResources.serviceAccounts.name | The name of Spark workload service account. | `spark` | +| workloadResources.serviceAccount.create | Whether to create a service account for Spark workload. | true | +| workloadResources.serviceAccount.name | The name of Spark workload service account. | `spark` | | workloadResources.labels | Labels to be applied for all workload resources. | `"app.kubernetes.io/component": "spark-workload"` | | workloadResources.annotations | Annotations to be applied for all workload resources. | `"helm.sh/resource-policy": keep` | | workloadResources.sparkApplicationSentinel.create | If enabled, sentinel resources will be created for operator to watch and reconcile for the health probe purpose. | false | | workloadResources.sparkApplicationSentinel.name | Name for sentinel resources. | "spark-app-sentinel" | -| workloadResources.sparkApplicationSentinel.sentinelNamespaces | A list of namespaces where sentinel resources will be created in. Note that these namespaces have to be a subset of `workloadResources.namespaces.data`. | | +| workloadResources.sparkApplicationSentinel.sentinelNamespaces.data | A list of namespaces where sentinel resources will be created in. Note that these namespaces have to be a subset of `workloadResources.namespaces.data`. | | | workloadResources.sparkClusterSentinel.create | If enabled, sentinel resources will be created for operator to watch and reconcile for the health probe purpose. | false | | workloadResources.sparkClusterSentinel.name | Name for sentinel resources. | "spark-cluster-sentinel" | -| workloadResources.sparkClusterSentinel.sentinelNamespaces | A list of namespaces where sentinel resources will be created in. Note that these namespaces have to be a subset of `workloadResources.namespaces.data`. | | +| workloadResources.sparkClusterSentinel.sentinelNamespaces.data | A list of namespaces where sentinel resources will be created in. Note that these namespaces have to be a subset of `workloadResources.namespaces.data`. | | | operatorConfiguration.append | If set to true, below conf file & properties would be appended to default conf. Otherwise, they would override default properties. | true | +| operatorConfiguration.configMap.annotations | Annotations to be applied for the operator configuration config map. | | +| operatorConfiguration.configMap.labels | Labels to be applied for the operator configuration config map. | | | operatorConfiguration.log4j2.properties | The default log4j2 configuration. | Refer default [log4j2.properties](../build-tools/helm/spark-kubernetes-operator/conf/log4j2.properties) | | operatorConfiguration.spark-operator.properties | The default operator configuration. | | | operatorConfiguration.metrics.properties | The default operator metrics (sink) configuration. | | | operatorConfiguration.dynamicConfig.create | If set to true, a config map would be created & watched by operator as source of truth for hot properties loading. | false | | operatorConfiguration.dynamicConfig.enabled | If set to true, operator would honor the created config map as source of truth for hot properties loading. | false | | operatorConfiguration.dynamicConfig.enable | Deprecated, use `operatorConfiguration.dynamicConfig.enabled`. Still honored: `enable: true` wins over `enabled: false`. Removed in chart 2.0.0 (SPARK-59533). | | +| operatorConfiguration.dynamicConfig.source | Source of the dynamic config: `configMap` (watched via informer, requires RBAC to read config maps) or `file` (periodically reloaded from the mounted config map). | `configMap` | | operatorConfiguration.dynamicConfig.annotations | Annotations to be applied for the dynamicConfig resources. | `"helm.sh/resource-policy": keep` | | operatorConfiguration.dynamicConfig.data | Data field (key-value pairs) that acts as hot properties in the config map. | `spark.kubernetes.operator.reconciler.intervalSeconds: "60"` | From 7bdd09dd57d9f57b6d408bfc605f8d6fc253b7e1 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Thu, 17 Sep 2026 03:01:02 -0700 Subject: [PATCH 2/2] Address comments --- docs/operations.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/operations.md b/docs/operations.md index 444d59856..766ab06b2 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -83,9 +83,11 @@ following table: | image.tag | The image tag of spark-kubernetes-operator. | 1.1.0-SNAPSHOT | | image.digest | The image digest of spark-kubernetes-operator. If set then it takes precedence and the image tag will be ignored. | | | imagePullSecrets | The image pull secrets of spark-kubernetes-operator. | | +| nameOverride | Override the chart name used in the names and labels of the created resources. | | | operatorDeployment.replicas | Operator replica count. Must be 1 unless leader election is configured. | 1 | | operatorDeployment.strategy.type | Operator pod upgrade strategy. Must be Recreate unless leader election is configured. | Recreate | | operatorDeployment.networkPolicy.enabled | When enabled, a NetworkPolicy allows ingress to the operator pod only on the health probe and metrics ports. Requires a CNI plugin with NetworkPolicy support. | false | +| operatorDeployment.networkPolicy.enable | Deprecated, use `operatorDeployment.networkPolicy.enabled`. Still honored: `enable: true` wins over `enabled: false`. Removed in chart 2.0.0 (SPARK-59533). | | | operatorDeployment.networkPolicy.metricsIngress | List of NetworkPolicyPeer(s) allowed to reach the metrics port, e.g. the Prometheus scraper. When empty, ingress to the metrics port is denied. | | | operatorDeployment.operatorPod.annotations | Custom annotations to be added to the operator pod | | | operatorDeployment.operatorPod.labels | Custom labels to be added to the operator pod | |