|
| 1 | +<!-- |
| 2 | +Licensed to the Apache Software Foundation (ASF) under one |
| 3 | +or more contributor license agreements. See the NOTICE file |
| 4 | +distributed with this work for additional information |
| 5 | +regarding copyright ownership. The ASF licenses this file |
| 6 | +to you under the Apache License, Version 2.0 (the |
| 7 | +"License"); you may not use this file except in compliance |
| 8 | +with the License. You may obtain a copy of the License at |
| 9 | +
|
| 10 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 11 | +
|
| 12 | +Unless required by applicable law or agreed to in writing, |
| 13 | +software distributed under the License is distributed on an |
| 14 | +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 15 | +KIND, either express or implied. See the License for the |
| 16 | +specific language governing permissions and limitations |
| 17 | +under the License. |
| 18 | +--> |
| 19 | + |
| 20 | +# Migration Guide |
| 21 | + |
| 22 | +This document lists the behavior changes of Apache Spark K8s Operator releases which may affect an |
| 23 | +existing deployment, and how to restore the previous behavior where possible. When upgrading across |
| 24 | +several releases, read every section in between. |
| 25 | + |
| 26 | +New features which are disabled by default are not listed, and neither are changes which only |
| 27 | +affect building the project, CI, tests or examples. |
| 28 | + |
| 29 | +## Upgrading from 1.0 to 1.1 |
| 30 | + |
| 31 | +### Kubernetes and CRDs |
| 32 | + |
| 33 | +- Since 1.1.0, K8s 1.35 or newer is recommended instead of 1.34 ([SPARK-59692](https://issues.apache.org/jira/browse/SPARK-59692)). |
| 34 | +- Since 1.1.0, the CRDs have `spec.suspend` for `SparkApplication` and `SparkCluster`, and the |
| 35 | + `Suspended` state for `SparkCluster`. Since `helm upgrade` does not update CRDs, replace them |
| 36 | + before upgrading the operator, e.g. with the CRDs of the Helm chart `1.9.0`, which deploys |
| 37 | + 1.1.0. Otherwise, the API server rejects or drops `spec.suspend`, and rejects the `Suspended` |
| 38 | + state ([SPARK-59475](https://issues.apache.org/jira/browse/SPARK-59475), [SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)). |
| 39 | + |
| 40 | + ```bash |
| 41 | + helm repo update |
| 42 | + helm pull spark/spark-kubernetes-operator --version 1.9.0 --untar |
| 43 | + kubectl replace -f spark-kubernetes-operator/crds/ |
| 44 | + ``` |
| 45 | + |
| 46 | +### Helm Chart |
| 47 | + |
| 48 | +- Since 1.1.0, `helm upgrade --reuse-values` from chart `1.8.0` fails, since it renders the new |
| 49 | + chart with the default values of chart `1.8.0`, which lack |
| 50 | + `operatorDeployment.networkPolicy.enabled`, `operatorConfiguration.dynamicConfig.enabled` and |
| 51 | + `operatorRbac.kueue` that the chart schema now requires. Use `--reset-then-reuse-values` of Helm |
| 52 | + 3.14 or newer, or pass the values file again with `-f` ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59519](https://issues.apache.org/jira/browse/SPARK-59519)). |
| 53 | +- Since 1.1.0, `operatorDeployment.networkPolicy.enable` and |
| 54 | + `operatorConfiguration.dynamicConfig.enable` are deprecated in favor of `enabled`, and |
| 55 | + `helm install` and `helm upgrade` print a warning for them. They are still honored until chart |
| 56 | + `2.0.0`: a feature is enabled when either key is `true`, so a legacy `enable: true` must be |
| 57 | + removed to disable it ([SPARK-59504](https://issues.apache.org/jira/browse/SPARK-59504), [SPARK-59540](https://issues.apache.org/jira/browse/SPARK-59540)). |
| 58 | +- Since 1.1.0, `operatorRbac.configManagement.create: false` takes effect. The chart then no longer |
| 59 | + creates the Role and RoleBinding for the `configMap` source of the dynamic config, which 1.0 |
| 60 | + created regardless. To restore the behavior before 1.1.0, set it to `true`, the default |
| 61 | + ([SPARK-59537](https://issues.apache.org/jira/browse/SPARK-59537)). |
| 62 | +- Since 1.1.0, the operator ClusterRole and Roles grant only `get`, `create` and `patch` on |
| 63 | + `events`, which are the verbs the operator uses. 1.0 also granted `list`, `watch`, `update` and |
| 64 | + `delete`. If other workloads use the operator ServiceAccount and need them, grant them with |
| 65 | + another Role or ClusterRole ([SPARK-59587](https://issues.apache.org/jira/browse/SPARK-59587)). |
| 66 | +- Since 1.1.0, each option of `operatorDeployment.operatorPod.operatorContainer.jvmArgs` is passed |
| 67 | + to the operator JVM as its own argument. 1.0 passed the whole value as one argument, which the |
| 68 | + JVM took as a single `-Dfile.encoding` system property, so the other options had no effect. The |
| 69 | + default options now take effect, e.g. the Parallel GC and a heap of 80% of the container memory |
| 70 | + limit (`2Gi` by default), fully touched at startup, instead of at most 25%. An |
| 71 | + `OutOfMemoryError` now crashes the JVM, so the container restarts. Keep a memory limit on the |
| 72 | + operator container, since the percentages apply to the node memory otherwise. The options also |
| 73 | + take precedence over `JAVA_TOOL_OPTIONS` and `JDK_JAVA_OPTIONS` in `operatorContainer.env`, and |
| 74 | + cannot contain a space, since the value is split on spaces. To restore the behavior before |
| 75 | + 1.1.0, set `jvmArgs` to `-Dfile.encoding=UTF8` ([SPARK-58426](https://issues.apache.org/jira/browse/SPARK-58426)). |
| 76 | + |
| 77 | +### Operator |
| 78 | + |
| 79 | +- Since 1.1.0, a non-positive `spark.kubernetes.operator.reconciler.retry.maxIntervalSeconds`, |
| 80 | + which is the default, leaves the interval between retries on reconciliation errors unlimited as |
| 81 | + documented. 1.0 capped it at about 15 seconds, so with the other defaults, the retries now span |
| 82 | + about 49 minutes instead of about 3 minutes. To restore the behavior before 1.1.0, set it to |
| 83 | + `15` ([SPARK-59892](https://issues.apache.org/jira/browse/SPARK-59892)). |
| 84 | + |
| 85 | +### SparkApplication |
| 86 | + |
| 87 | +- Since 1.1.0, when the operator retries creating the driver resources of an application, it waits |
| 88 | + as long as the `Retry-After` hint of the API server asks, capped at |
| 89 | + `spark.kubernetes.operator.api.secondaryResourceCreateMaxBackoffMillis`. 1.0 ignored the hint, |
| 90 | + and retried errors other than `409` and `429`, e.g. `503`, right away. A request which the API |
| 91 | + server never answered, e.g. on a connection reset or a timeout, is retried with exponential |
| 92 | + backoff too, while 1.0 failed the attempt with `SchedulingFailure` at once. The number of |
| 93 | + attempts is still bounded by `spark.kubernetes.operator.api.secondaryResourceCreateMaxAttempts` |
| 94 | + ([SPARK-59092](https://issues.apache.org/jira/browse/SPARK-59092), [SPARK-59694](https://issues.apache.org/jira/browse/SPARK-59694)). |
| 95 | +- Since 1.1.0, the duration of an attempt which is compared with `restartCounterResetMillis` is |
| 96 | + measured from the first state after `Submitted` or `ScheduledToRestart`, normally |
| 97 | + `DriverRequested`, instead of from `Submitted` or `ScheduledToRestart`. The restart backoff no |
| 98 | + longer counts, so an attempt which reset the restart counters in 1.0 may not reset them, and the |
| 99 | + application may reach `maxRestartAttempts` earlier. To keep a similar threshold, decrease |
| 100 | + `restartCounterResetMillis` by the restart backoff, e.g. `restartBackoffMillis` |
| 101 | + ([SPARK-59542](https://issues.apache.org/jira/browse/SPARK-59542)). |
| 102 | +- Since 1.1.0, an application which is configured to restart ends in `ResourceReleased` instead of |
| 103 | + `TerminatedWithoutReleaseResources` after its last attempt, even if its `resourceRetainPolicy` |
| 104 | + retains the resources, since they are released at the end of every attempt. So |
| 105 | + `resourceRetainDurationMillis` no longer applies to it, while `ttlAfterStopMillis` still deletes |
| 106 | + it at the same time as in 1.0. A tool which waits for `TerminatedWithoutReleaseResources` has to |
| 107 | + wait for `ResourceReleased` instead ([SPARK-59732](https://issues.apache.org/jira/browse/SPARK-59732)). |
| 108 | +- Since 1.1.0, an application whose driver pod fails before any of its containers starts, e.g. |
| 109 | + when the pod is evicted or its node is deleted, moves to `Failed`, or `DriverEvicted` if it is |
| 110 | + evicted, right away. 1.0 kept it in `DriverRequested` until `DriverStartTimedOut`, or forever if |
| 111 | + `driverStartTimeoutMillis` was disabled. Since `DriverStartTimedOut` is an infrastructure failure |
| 112 | + while the others are not, `restartPolicy: OnInfrastructureFailure` no longer restarts such an |
| 113 | + application. Use `OnFailure` to restart it ([SPARK-59729](https://issues.apache.org/jira/browse/SPARK-59729)). |
| 114 | + |
| 115 | +### SparkCluster |
| 116 | + |
| 117 | +The changes of the master and worker resources below apply when the operator creates them, i.e. to |
| 118 | +a `SparkCluster` created after the upgrade. A running `SparkCluster` keeps its resources until it |
| 119 | +is recreated, e.g. by suspending and resuming it with [`spec.suspend`](spark_custom_resources.md#suspend). |
| 120 | + |
| 121 | +- Since 1.1.0, the workers start the external shuffle service by default, so that dynamic |
| 122 | + allocation works out of the box. To restore the behavior before 1.1.0, set |
| 123 | + `spark.shuffle.service.enabled` to `false` in `spec.sparkConf` ([SPARK-58835](https://issues.apache.org/jira/browse/SPARK-58835)). |
| 124 | +- Since 1.1.0, the NetworkPolicy of the workers also admits ingress from every driver pod |
| 125 | + (`spark-role: driver`) in the namespace, so that the driver of an application running on the |
| 126 | + cluster can fetch task results larger than `spark.task.maxDirectResultSize` from the executors. |
| 127 | + 1.0 admitted only the pods of the cluster ([SPARK-58649](https://issues.apache.org/jira/browse/SPARK-58649)). |
| 128 | +- Since 1.1.0, the `terminationGracePeriodSeconds` of the master and worker pod templates is |
| 129 | + respected. 1.0 always overwrote it with `0`, which is still the default. To restore the behavior |
| 130 | + before 1.1.0, remove it from the pod templates ([SPARK-59795](https://issues.apache.org/jira/browse/SPARK-59795)). |
| 131 | +- Since 1.1.0, the `dnsConfig` of the worker pod template is kept, and the domain of the worker |
| 132 | + service is appended to its `searches`. 1.0 replaced the whole `dnsConfig`. To restore the |
| 133 | + behavior before 1.1.0, remove `dnsConfig` from the worker pod template ([SPARK-59797](https://issues.apache.org/jira/browse/SPARK-59797)). |
| 134 | +- Since 1.1.0, a cluster whose master and worker resources cannot be requested because of a |
| 135 | + failure which may clear on its own, e.g. a timeout, a `429`, a `500` or a request which the API |
| 136 | + server never answered, stays `Submitted` and is retried at the next reconciliation, instead of |
| 137 | + failing with `SchedulingFailure` and then `Failed`. A request which the API server rejected |
| 138 | + still fails the cluster ([SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)). |
| 139 | +- Since 1.1.0, the message of the `Failed` state after `SchedulingFailure` is |
| 140 | + `Cluster failed, since requesting its resources failed. See the preceding SchedulingFailure state for the cause.` |
| 141 | + instead of `Cannot process cluster status.` ([SPARK-59870](https://issues.apache.org/jira/browse/SPARK-59870)). |
| 142 | + |
| 143 | +### Docker Image |
| 144 | + |
| 145 | +- Since 1.1.0, the image is based on `alpine:3.24` instead of `alpine:3.23`, and it does not |
| 146 | + install `libstdc++`, which the operator does not use. An image built on top of it which needs |
| 147 | + the package has to install it ([SPARK-59834](https://issues.apache.org/jira/browse/SPARK-59834), [SPARK-59836](https://issues.apache.org/jira/browse/SPARK-59836)). |
| 148 | + |
| 149 | + ```dockerfile |
| 150 | + FROM apache/spark-kubernetes-operator:1.1.0 |
| 151 | + USER root |
| 152 | + RUN apk add --no-cache libstdc++ |
| 153 | + USER spark |
| 154 | + ``` |
| 155 | + |
| 156 | +### Java API |
| 157 | + |
| 158 | +- Since 1.1.0, `org.apache.spark.k8s.operator.Constants` is `final` and has a private constructor, |
| 159 | + so it cannot be extended or instantiated ([SPARK-59679](https://issues.apache.org/jira/browse/SPARK-59679)). |
| 160 | +- Since 1.1.0, `ApplicationStatus.terminateOrRestart(RestartConfig, ResourceRetainPolicy, String, boolean)` |
| 161 | + is deprecated for removal. It ignores `resourceRetainPolicy`, so it no longer returns |
| 162 | + `TerminatedWithoutReleaseResources`. Use `terminateOrRestart(RestartConfig, String, boolean)` |
| 163 | + instead ([SPARK-59734](https://issues.apache.org/jira/browse/SPARK-59734)). |
| 164 | +- Since 1.1.0, `ClusterStateSummary` has the `Suspended` constant, so an exhaustive `switch` over |
| 165 | + it needs a case for it ([SPARK-59750](https://issues.apache.org/jira/browse/SPARK-59750)). |
0 commit comments