[SPARK-59991] Remove unused operatorConfiguration.append and fix broken conf/ links

### What changes were proposed in this pull request?

This PR aims to remove the unused `operatorConfiguration.append` from the Helm chart and fix the broken `conf/` links in the documentation.

### Why are the changes needed?

The Helm chart has never had a `conf/` directory, so `operatorConfiguration.append` has no effect in any release, e.g. [1.0.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0) (2026-07-23). However, `values.yaml` and the documentation still describe it and link to the non-existent conf files.

### Does this PR introduce _any_ user-facing change?

Yes, `operatorConfiguration.append` is removed, but the rendered manifests are identical. The migration guide is updated.

### How was this patch tested?

Pass the CIs. I also verified manually that `helm template` renders identical manifests before and after this PR.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

Closes #937 from dongjoon-hyun/SPARK-59991.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Author: dongjoon-hyun

build-tools/helm/spark-kubernetes-operator/templates/spark-operator.yaml

@@ -218,15 +218,9 @@ metadata:
   {{- end }}
 data:
   log4j2.properties: |
-{{- if .Values.operatorConfiguration.append }}
-{{-   with $.Files.Get "conf/log4j2.properties" }}{{ . | nindent 4 }}{{ end }}
-{{- end }}
 {{- with index .Values.operatorConfiguration "log4j2.properties" }}{{ . | nindent 4 }}{{ end }}
   spark-operator.properties: |
 {{- include "spark-operator.defaultPropertyOverrides" . | nindent 4 }}
-{{- if .Values.operatorConfiguration.append }}
-{{-   with $.Files.Get "conf/spark-operator.properties" }}{{ . | nindent 4 }}{{ end }}
-{{- end }}
 {{- with index .Values.operatorConfiguration "spark-operator.properties" }}{{ . | nindent 4 }}{{ end }}
   metrics.properties: |
 {{- with index .Values.operatorConfiguration "metrics.properties" }}{{ . | nindent 4 }}{{ end }}

build-tools/helm/spark-kubernetes-operator/values.schema.json

@@ -792,14 +792,7 @@
     "operatorConfiguration": {
       "type": "object",
       "description": "Operator configuration",
-      "required": [
-        "append"
-      ],
       "properties": {
-        "append": {
-          "type": "boolean",
-          "description": "Append configuration instead of replacing"
-        },
         "configMap": {
           "type": "object",
           "description": "Configuration for the operator config map",

build-tools/helm/spark-kubernetes-operator/values.yaml

@@ -209,9 +209,6 @@ workloadResources:
     "app.kubernetes.io/component": "spark-workload"
 
 operatorConfiguration:
-  # If set to true, below properties would be appended to default conf files under conf/
-  # Otherwise, below would override default conf files
-  append: true
   # Labels and annotations for the operatorConfiguration config map
   configMap:
     annotations:

docs/configuration.md

@@ -23,9 +23,8 @@ under the License.
 
 Spark Operator supports different ways to configure the behavior:
 
-* **spark-operator.properties** provided when deploying the operator. In addition to the
-  [property file](../build-tools/helm/spark-kubernetes-operator/conf/spark-operator.properties),
-  it is also possible to override or append config properties in helm
+* **spark-operator.properties** provided when deploying the operator. It is possible to override
+  the [default values](./config_properties.md) of config properties in helm
   [Values files](../build-tools/helm/spark-kubernetes-operator/values.yaml).
 * **System Properties** : when provided as system properties (e.g. via -D options to the
   operator JVM), they apply only to the properties which are not provided in the property file,

docs/migration_guide.md

@@ -93,6 +93,12 @@ affect building the project, CI, tests or examples.
   take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`, and
   cannot contain a space, since the value is split on spaces. To restore the behavior before
   1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)).
+- Since 1.1.0, `operatorConfiguration.append` is removed, and the chart no longer reads
+  `conf/log4j2.properties` and `conf/spark-operator.properties` in the chart directory, which it
+  has never included. A values file which still sets `append` keeps working, since the key is
+  ignored. If you added such files to the chart, prepend their content to
+  `operatorConfiguration.log4j2.properties` and `operatorConfiguration.spark-operator.properties`
+  respectively to restore the behavior before 1.1.0.
 
 ### Operator
 

docs/operations.md

@@ -162,10 +162,9 @@ following table:
 | workloadResources.sparkClusterSentinel.create                    | If enabled, sentinel resources will be created for operator to watch and reconcile for the health probe purpose.                                                               | false                                                                                                   |
 | workloadResources.sparkClusterSentinel.name                      | Name for sentinel resources.                                                                                                                                                   | "spark-cluster-sentinel"                                                                                    |
 | workloadResources.sparkClusterSentinel.sentinelNamespaces.data   | A list of namespaces where sentinel resources will be created in. Note that these namespaces have to be a subset of `workloadResources.namespaces.data`.                       |                                                                                                         |
-| operatorConfiguration.append                                     | If set to true, below conf file & properties would be appended to default conf. Otherwise, they would override default properties.                                             | true                                                                                                    |
 | operatorConfiguration.configMap.annotations                      | Annotations to be applied for the operator configuration config map.                                                                                                           |                                                                                                         |
 | operatorConfiguration.configMap.labels                           | Labels to be applied for the operator configuration config map.                                                                                                                |                                                                                                         |
-| operatorConfiguration.log4j2.properties                          | The default log4j2 configuration.                                                                                                                                              | Refer default [log4j2.properties](../build-tools/helm/spark-kubernetes-operator/conf/log4j2.properties) |
+| operatorConfiguration.log4j2.properties                          | The default log4j2 configuration.                                                                                                                                              | Refer default [values.yaml](../build-tools/helm/spark-kubernetes-operator/values.yaml)                  |
 | operatorConfiguration.spark-operator.properties                  | The default operator configuration.                                                                                                                                            |                                                                                                         |
 | operatorConfiguration.metrics.properties                         | The default operator metrics (sink) configuration.                                                                                                                             |                                                                                                         |
 | operatorConfiguration.dynamicConfig.create                       | If set to true, a config map would be created & watched by operator as source of truth for hot properties loading.                                                             | false                                                                                                   |

tests/e2e/helm/helm-test-values/configmap-metadata/values.yaml

@@ -26,7 +26,6 @@
 # and can be used to validate the Helm test for this functionality.
 
 operatorConfiguration:
-  append: true
   configMap:
     labels:
       test.example.com/label: "test-value"