From 39a58510545e908cf5d72c727f44616d04549f7c Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Fri, 7 Feb 2025 11:26:11 -0500 Subject: [PATCH 01/13] Re-write parts of AEP-4016 to align with the new KEP changes --- .../4016-in-place-updates-support/README.md | 168 ++++++++++-------- 1 file changed, 93 insertions(+), 75 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 0a840c0d255..09ecb5eba35 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -17,80 +17,125 @@ ## Summary - VPA applies its recommendations with a mutating webhook, during pod creation. It can also evict -pods expecting that it will apply the recommendation when the pod is recreated. This is a -disruptive process so VPA has some mechanism to avoid too frequent disruptive updates. +pods expecting that it will apply the recommendation when the pod is recreated. Today, this process +is very disruptive as any changes in recommendations requires a pod to be recreated. + +We can instead reduce the amount of disruption by leveraging the [in-place update feature] which is +currently an [alpha feature since 1.27] and graduating to [beta in 1.33]. -This proposal allows VPA to apply its recommendations more frequently, with less disruption by -using the -[in-place update feature] which is an alpha feature [available in Kubernetes 1.27.] This proposal enables only core uses -of in place updates in VPA with intention of gathering more feedback. Any more advanced uses of in place updates in VPA -(like applying different recommendations during pod initialization) will be introduced as separate enhancement -proposals. +This proposal enables only core uses of in place updates in VPA with intention of providing the +foundational pieces. Further advanced uses of in place updates in VPA (like applying different +recommendations during pod initialization or providing more frequent smaller updates) will be +introduced as separate enhancement proposals. [in-place update feature]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources -[available in Kubernetes 1.27.]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#api-change-3 +[alpha feature since 1.27]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#api-change-3 +[beta in 1.33]: https://github.com/orgs/kubernetes/projects/178/views/1 + +### A Note On Disruptions {#disruptions} + +It is important to note that **VPA cannot guarantee NO disruptions**. This is because the +underlying container runtime is responsible for actuating the resize operation and there are no +guarantees provided (see [this thread]for more information). + +This proposal therefore focuses on *reducing* disruptions while still harnessing the benefits of +VPA. + +[this thread]: https://github.com/kubernetes/autoscaler/issues/7722#issue-2796215055 ### Goals -* Allow VPA to actuate without disruption, -* Allow VPA to actuate more frequently, when it can actuate without disruption, -* Allow VPA to actuate in situations where actuation by eviction doesn't work. +* Allow VPA to actuate with reduced disruption. +* Allow VPA to actuate more frequently. +* Allow VPA to actuate in situations where actuation by eviction is not desirable. ### Non-Goals +* Allow VPA to operate with NO disruptions, see the [note above](#disruptions). * Improved handling of injected sidecars * Separate AEP will improve VPAs handling of injected sidecars. ## Proposal -Add new supported values of [`UpdateMode`]: +Add a new supported value of [`UpdateMode`]: -* `InPlaceOnly` and -* `InPlaceOrRecreate`. +* `InPlaceOrRecreate` + +Here we specify `OrRecreate` to make sure the user explicitly knows that the container may be +restarted. [`UpdateMode`]: https://github.com/kubernetes/autoscaler/blob/71b489f5aec3899157b37472cdf36a1de223d011/vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go#L124 ## Context -[In-place update of pod resources KEP] is available in alpha in [Kubernetes 1.27]. The feature allows changing container -resources while the container is running. It also adds [`ResizePolicy`] field to Container. This field indicates for an -individual resource if a container needs to be restarted by kubelet when the resource is changed. For example it may be -the case that a Container automatically adapts to a change in CPU, but needs to be restarted for a change in Memory to -take effect. +[In-place update of pod resources KEP] is available in alpha in 1.27 and graduating to beta in +1.33. The feature allows changing container resources while the container is running. It adds +two key features: + +* A [`/resize` subresource] that can be used to mutate the `Pod.Spec.Containers[i].Resources` + field. +* A [`ResizePolicy`] field to Container. This field allows to the user to specify the behavior when + modifying a resource value. Currently it has two modes: + - `NotRequired` (default) which indicates to the container runtime that it should try to resize + the container without restarting. However, it does not guarantee that a restart will not + happen. + - `RestartContainer` which indicates that any mutation to the resource requires a restart (for + example, this is important for Java apps using the `-xmxN` which are unable to resize memory + without restarting). + +Note that resize operations will NOT change the pod's quality of service (QoS) class. [In-place update of pod resources KEP]: https://github.com/kubernetes/enhancements/issues/1287 -[Kubernetes 1.27]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#api-change-3 -[`ResizePolicy`]: https://github.com/kubernetes/api/blob/8360d82aecbc72aa039281a394ebed2eaf0c0ccc/core/v1/types.go#L2448 +[`/resize` subresource]:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources#api-changes +[`ResizePolicy`]: https://github.com/kubernetes/api/blob/4dccc5e86b957cea946a63c4f052ee7dec3946ce/core/v1/types.go#L2636 ## Design Details -In the update modes existing before (`Initial` and `Recreate`) only admission controller was changing pod spec (updater -was responsible for evicting pods so that admission controller could change them during admission). +Today, only the VPA admission controller is responsible for changing the pod spec. + +The VPA updater is responsible for evicting pods so that the admission controller can change them +during admission. + +In the newly added `InPlaceOrRecreate` mode, the VPA Updater will attempt to execute in-plac +updates FIRST. In certain situations, if it is unable to process an in-place update in time, it +will evict the pod to force a change. -In the newly added `InPlaceOnly` and `InPlaceOrRecreate` modes VPA Updater will execute in-place updates (which change -pod spec without involving admission controller). +We classify three types of updates in the context of this new mode: -### Applying Updates During Pod Admission +1. Updates on pod admission +2. Disruption-free updates +3. Disruptive updates -For VPAs in `InPlaceOnly` and `InPlaceOrRecreate` modes VPA Admission Controller will apply updates to starting pods, -like it does for VPAs in `Initial`, `Auto`, and `Recreate` modes. +**NOTE:** Here "disruptive" is from the perspective of VPA. As mentioned above, the container +runtime MAY choose to restart a container or pod as needed. -### Applying Disruption-free Updates +More on these two types of changes below. -When an update only changes resources for which a container indicates that it doesn't require a restart, then VPA can -attempt to actuate the change without disrupting the pod. VPA Updater will: +### 1. Applying Updates During Pod Admission + +For VPAs using the new `InPlaceOrRecreate` mode, the VPA Admission Controller will apply updates to +starting pods just as it does for VPAs in `Initial`, `Auto`, and `Recreate` modes. + +### 2. Applying Disruption-free Updates (**NEW**) + +In the `InPlaceOrRecreate` mode when an update only changes resources for which a container +indicates that it doesn't require a restart, then VPA can attempt to actuate the change without +disrupting the pod. VPA Updater will: * attempt to actuate such updates in place, * attempt to actuate them if difference between recommendation and request is at least 10% * even if pod has been running for less than 12h, * if necessary execute only partial updates (if some changes would require a restart). -### Applying Disruptive Updates +If VPA updater fails to update the size of a container, we ignore the failure and try again. We +will not intentionally evict the pod unless the conditions call for a disruptive update (see +below). -In both `InPlaceOnly` and `InPlaceOrRecreate` modes VPA updater will attempt to apply updates that require container -restart in place. It will update them under conditions that would trigger update with `Recreate` mode. That is it will -apply disruptive updates if: +### 3. Applying Disruptive Updates + +In the `InPlaceOrRecreate` modes VPA updater will attempt to apply updates that require container +restart in place. It will update them under the same conditions that would trigger update with +`Recreate` mode. That is it will apply disruptive updates if: * Any container has a request below the corresponding `LowerBound` or * Any container has a request above the corresponding `UpperBound` or @@ -98,7 +143,7 @@ apply disruptive updates if: running undisrupted for at least 12h. * Successful disruptive update counts as disruption here (and prevents further disruptive updates to the pod for 12h). -In `InPlaceOrRecreate` mode (but not in `InPlaceOnly` mode) VPA updater will evict pod to actuate a recommendation if it +For disruptive updates, the VPA updater will evict a pod to actuate a recommendation if it attempted to apply the recommendation in place and failed. VPA updater will consider that the update failed if: @@ -107,13 +152,7 @@ VPA updater will consider that the update failed if: * The pod has `.status.resize: InProgress` and more than 1 hour elapsed since the update: * There seems to be a bug where containers that say they need to be restarted get stuck in update, hopefully it gets fixed and we don't have to worry about this by beta. -* Patch attempt will return an error. - * If the attempt fails because it would change pods QoS: - * `InPlaceOrRecreate` will treat it as any other failure and consider evicting the pod. - * `InPlaceOnly` will consider applying request slightly lower than the limit. - -Those failure modes shouldn't disrupt pod operations, only update. If there are problems that can disrupt pod operation -we should consider not implementing the `InPlaceOnly` mode. +* Patch attempt returns an error. ### Comparison of `UpdateMode`s @@ -133,7 +172,7 @@ VPA updater considers the following conditions when deciding if it should apply recommendation or - At least one container has at least one resource request higher than the upper bound of the corresponding recommendation. -- Disruption-free update - doesn't change any resources for which the relevant container specifies +- **NEW** Disruption-free update - doesn't change any resources for which the relevant container specifies `RestartPolicy: RestartContainer`. `Auto` / `Recreate` evicts pod if: @@ -142,18 +181,18 @@ VPA updater considers the following conditions when deciding if it should apply * Outside recommended range, * Long-lived pod with significant change. -`InPlaceOnly` and `InPlaceOrRecreate` will attempt to apply a disruption-free update in place if it meets at least one +`InPlaceOrRecreate` will attempt to apply a disruption-free update in place if it meets at least one of the following conditions: * Quick OOM, * Outside recommended range, * Significant change. -`InPlaceOnly` and `InPlaceOrRecreate` when considering a disruption-free update in place ignore some conditions that +`InPlaceOrRecreate` when considering a disruption-free update in place ignore some conditions that influence eviction decission in the `Recreate` mode: * [`CanEvict`] won't be checked and * Pods with significant change can be updated even if they are not long-lived. -`InPlaceOnly` and `InPlaceOrRecreate` will attempt to apply updates that are **not** disruption-free in place under +`InPlaceOrRecreate` will attempt to apply updates that are **not** disruption-free in place under the same conditions that apply to updates in the `Recreate` mode. `InPlaceOrRecreate` will attempt to apply updates that by eviction when: @@ -169,41 +208,19 @@ the same conditions that apply to updates in the `Recreate` mode. ### Test Plan -The following test scenarios will be added to e2e tests. Both `InPlaceOnly` and `InPlaceOrRecreate` modes will be tested -and they should behave the same: +The following test scenarios will be added to e2e tests. The `InPlaceOrRecreate` mode will be +tested in the following scenarios: * Admission controller applies recommendation to pod controlled by VPA. * Disruption-free in-place update applied to all containers of a pod (request in recommendation bounds). * Partial disruption-free update applied to some containers of a pod, some disruptive changes skipped (request in recommendation bounds). * Disruptive in-place update applied to all containers of a pod (request out ouf recommendation bounds). - -There will be also scenarios testing differences between `InPlaceOnly` and `InPlaceOrRecreate` modesL -* Disruptive in-place update will fail. In `InPlaceOnly` pod should not be evicted, in the `InPlaceOrRecreate` pod - should be evicted and the recommendation applied. -* VPA attempts an update that would change Pods QoS (`RequestsOnly` scaling, request initially < limit, recommendation - equal to limit). In `InPlaceOnly` pod should not be evicted, request slightly lower than the recommendation will be - applied. In the `InPlaceOrRecreate` pod should be evicted and the recommendation applied. +* Disruptive in-place update will fail. Pod should be evicted and the recommendation applied. +* Disruption free in-place updates fail, pod should not be evicted. ### Details still to consider -#### Ensure in-place resize request doesn't cause restarts - -Currently the container [resize policy](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/#container-resize-policies) -can be either `NotRequired` or `RestartContainer`. With `NotRequired` in-place update could still end up -restarting the container if in-place update is not possible, depending on kubelet and container -runtime implementation. However in the proposed design it should be VPA's decision whether to fall back -to restarts or not. - -Extending or changing the existing API for in-place updates is possible, e.g. adding a new -`MustNotRestart` container resize policy. - -#### Should `InPlaceOnly` mode be dropped - -The use case for `InPlaceOnly` is not understood yet. Unless we have a strong signal it solves real -needs we should not implement it. Also VPA cannot ensure no restart would happen unless -*Ensure in-place resize request doesn't cause restarts* (see above) is solved. - #### Careful with memory scale down Downsizing memory may have to be done slowly to prevent OOMs if application starts to allocate rapidly. @@ -212,3 +229,4 @@ Needs more research on how to scale down on memory safely. ## Implementation History - 2023-05-10: initial version +- 2025-02-07: Updates to align with latest changes to [KEP-1287](https://github.com/kubernetes/enhancements/issues/1287). From 10e2339cf2f32b544696e55a4a888d7feaa44305 Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Fri, 7 Feb 2025 23:17:41 +0000 Subject: [PATCH 02/13] Small edits based on comments --- .../enhancements/4016-in-place-updates-support/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 09ecb5eba35..5d949141243 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -37,7 +37,9 @@ introduced as separate enhancement proposals. It is important to note that **VPA cannot guarantee NO disruptions**. This is because the underlying container runtime is responsible for actuating the resize operation and there are no -guarantees provided (see [this thread]for more information). +guarantees provided (see [this thread]for more information). However, in practice if the +underlying container runtime supports it, we expect these disruptions to be minimal and that MOST +of the time the updates will be done in-place. This proposal therefore focuses on *reducing* disruptions while still harnessing the benefits of VPA. @@ -62,7 +64,7 @@ Add a new supported value of [`UpdateMode`]: * `InPlaceOrRecreate` -Here we specify `OrRecreate` to make sure the user explicitly knows that the container may be +Here we specify `OrRecreate` to make sure the user explicitly knows that the pod may be restarted. [`UpdateMode`]: https://github.com/kubernetes/autoscaler/blob/71b489f5aec3899157b37472cdf36a1de223d011/vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go#L124 @@ -77,7 +79,7 @@ two key features: field. * A [`ResizePolicy`] field to Container. This field allows to the user to specify the behavior when modifying a resource value. Currently it has two modes: - - `NotRequired` (default) which indicates to the container runtime that it should try to resize + - `PreferNoRestart` (default) which indicates to the container runtime that it should try to resize the container without restarting. However, it does not guarantee that a restart will not happen. - `RestartContainer` which indicates that any mutation to the resource requires a restart (for From a769eb1547e4fcf3608fa3c6f1b179d39f85502b Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Mon, 10 Feb 2025 21:16:43 -0500 Subject: [PATCH 03/13] Update vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md Co-authored-by: Adrian Moisey --- .../enhancements/4016-in-place-updates-support/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 5d949141243..8d61c3bff50 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -99,7 +99,7 @@ Today, only the VPA admission controller is responsible for changing the pod spe The VPA updater is responsible for evicting pods so that the admission controller can change them during admission. -In the newly added `InPlaceOrRecreate` mode, the VPA Updater will attempt to execute in-plac +In the newly added `InPlaceOrRecreate` mode, the VPA Updater will attempt to execute in-place updates FIRST. In certain situations, if it is unable to process an in-place update in time, it will evict the pod to force a change. From 123420974c74e0b69c6f5a438113a9157d54d22a Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Mon, 10 Feb 2025 21:16:54 -0500 Subject: [PATCH 04/13] Update vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md Co-authored-by: Adrian Moisey --- .../enhancements/4016-in-place-updates-support/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 8d61c3bff50..0d2da4eb425 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -37,7 +37,7 @@ introduced as separate enhancement proposals. It is important to note that **VPA cannot guarantee NO disruptions**. This is because the underlying container runtime is responsible for actuating the resize operation and there are no -guarantees provided (see [this thread]for more information). However, in practice if the +guarantees provided (see [this thread] for more information). However, in practice if the underlying container runtime supports it, we expect these disruptions to be minimal and that MOST of the time the updates will be done in-place. From 1c3006af5e9e06db2feec29c4394b3cfbf84da2d Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Mon, 10 Feb 2025 21:17:02 -0500 Subject: [PATCH 05/13] Update vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md Co-authored-by: Adrian Moisey --- .../enhancements/4016-in-place-updates-support/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 0d2da4eb425..e76fc608fc3 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -19,7 +19,7 @@ VPA applies its recommendations with a mutating webhook, during pod creation. It can also evict pods expecting that it will apply the recommendation when the pod is recreated. Today, this process -is very disruptive as any changes in recommendations requires a pod to be recreated. +is potentially disruptive as any changes in recommendations requires a pod to be recreated. We can instead reduce the amount of disruption by leveraging the [in-place update feature] which is currently an [alpha feature since 1.27] and graduating to [beta in 1.33]. From cc4ac5fa01053eb8096fbb1064bb6ae0b4b7a381 Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Wed, 19 Feb 2025 19:40:06 +0000 Subject: [PATCH 06/13] Edits to AEP as discussed over slack to align with current VPA behavior --- .../4016-in-place-updates-support/README.md | 105 +++++++++--------- 1 file changed, 51 insertions(+), 54 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index e76fc608fc3..d8036749e11 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -7,9 +7,8 @@ - [Proposal](#proposal) - [Context](#context) - [Design Details](#design-details) - - [Applying Updates During Pod Admission](#applying-updates-during-pod-admission) - - [Applying Disruption-free Updates](#Applying-disruption-free-updates) - - [Applying Disruptive Updates](#applying-disruptive-updates) + - [1. Applying Updates During Pod Admission](#pod-admission) + - [2. In-Place Updates (**NEW**)](#in-place) - [Comparison of `UpdateMode`s](#comparison-of-updatemodes) - [Test Plan](#test-plan) - [Implementation History](#implementation-history) @@ -49,11 +48,11 @@ VPA. ### Goals * Allow VPA to actuate with reduced disruption. -* Allow VPA to actuate more frequently. * Allow VPA to actuate in situations where actuation by eviction is not desirable. ### Non-Goals +* Allow VPA to actuate more frequently. * Allow VPA to operate with NO disruptions, see the [note above](#disruptions). * Improved handling of injected sidecars * Separate AEP will improve VPAs handling of injected sidecars. @@ -69,6 +68,13 @@ restarted. [`UpdateMode`]: https://github.com/kubernetes/autoscaler/blob/71b489f5aec3899157b37472cdf36a1de223d011/vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go#L124 +For the initial release of in-place updates with VPA, in-place updates will only be available +using the `InPlaceOrRecreate` mode. In the future, once the SIG feels that the feature is +mature enough, this behavior will become the default behavior for the `Auto` mode. See the [`Auto` +mode documentation]. + +[`Auto` mode documentation]: https://github.com/kubernetes/autoscaler/blob/78c8173b979316f892327022d53369760b000210/vertical-pod-autoscaler/docs/api.md#updatemode + ## Context [In-place update of pod resources KEP] is available in alpha in 1.27 and graduating to beta in @@ -100,57 +106,55 @@ The VPA updater is responsible for evicting pods so that the admission controlle during admission. In the newly added `InPlaceOrRecreate` mode, the VPA Updater will attempt to execute in-place -updates FIRST. In certain situations, if it is unable to process an in-place update in time, it -will evict the pod to force a change. +updates FIRST. If it is unable to process an in-place update in time, it will evict the pod to force +a change. -We classify three types of updates in the context of this new mode: +This will effectively match the current behavior in `Auto` except that resizes will first be +attempted in-place. -1. Updates on pod admission -2. Disruption-free updates -3. Disruptive updates +In the future, this logic may be improved to: +* Provide more frequent resizes. +* Make changes that are only attempted using in-place resizes and wouldn't ultimately result in an + eviction on failure. +* In the case of failure, make smaller updates to circumvent a node that does not have enough + headroom to accept the full resize but could accomodate a smaller one. -**NOTE:** Here "disruptive" is from the perspective of VPA. As mentioned above, the container -runtime MAY choose to restart a container or pod as needed. +We classify two types of updates in the context of this new mode: -More on these two types of changes below. +1. Updates on pod admission +2. In-place updates -### 1. Applying Updates During Pod Admission +### 1. Applying Updates During Pod Admission {#pod-admission} For VPAs using the new `InPlaceOrRecreate` mode, the VPA Admission Controller will apply updates to starting pods just as it does for VPAs in `Initial`, `Auto`, and `Recreate` modes. -### 2. Applying Disruption-free Updates (**NEW**) - -In the `InPlaceOrRecreate` mode when an update only changes resources for which a container -indicates that it doesn't require a restart, then VPA can attempt to actuate the change without -disrupting the pod. VPA Updater will: -* attempt to actuate such updates in place, -* attempt to actuate them if difference between recommendation and request is at least 10% - * even if pod has been running for less than 12h, -* if necessary execute only partial updates (if some changes would require a restart). - -If VPA updater fails to update the size of a container, we ignore the failure and try again. We -will not intentionally evict the pod unless the conditions call for a disruptive update (see -below). - -### 3. Applying Disruptive Updates +### 2. In-Place Updates (**NEW**) {#in-place} In the `InPlaceOrRecreate` modes VPA updater will attempt to apply updates that require container -restart in place. It will update them under the same conditions that would trigger update with -`Recreate` mode. That is it will apply disruptive updates if: +restart in-place. It will update them under the same conditions that would trigger update with +`Recreate` mode. That is it will apply an in-place update if: * Any container has a request below the corresponding `LowerBound` or * Any container has a request above the corresponding `UpperBound` or * Difference between sum of pods requests and sum of recommendation `Target`s is more than 10% and the pod has been running undisrupted for at least 12h. - * Successful disruptive update counts as disruption here (and prevents further disruptive updates to the pod for 12h). + * NOTE: A successful update counts as disruption here (and prevents further disruptive updates to the pod for 12h). + +(NEW!) In addition, VPA will attempt an in-place update in some cases where we NORMALLY would not +be able to perform an eviction, including: + +* If `CanEvict` is false. +* If any of the `EvictionRequirements` on the VPA are not true. -For disruptive updates, the VPA updater will evict a pod to actuate a recommendation if it -attempted to apply the recommendation in place and failed. +These additional resizes can be attempted because the eviction fallback would fail anyway. + +The VPA updater will evict a pod to actuate a recommendation if it attempted to apply the +recommendation in place and failed. VPA updater will consider that the update failed if: * The pod has `.status.resize: Infeasible` or -* The pod has `.status.resize: Deferred` and more than 1 minute elapsed since the update or +* The pod has `.status.resize: Deferred` and more than 5 minutes elapsed since the update or * The pod has `.status.resize: InProgress` and more than 1 hour elapsed since the update: * There seems to be a bug where containers that say they need to be restarted get stuck in update, hopefully it gets fixed and we don't have to worry about this by beta. @@ -158,7 +162,7 @@ VPA updater will consider that the update failed if: ### Comparison of `UpdateMode`s -VPA updater considers the following conditions when deciding if it should apply an update: +Today, VPA updater considers the following conditions when deciding if it should apply an update: - [`CanEvict`]: - Pod is `Pending` or - There are enough running pods in the controller. @@ -182,24 +186,16 @@ VPA updater considers the following conditions when deciding if it should apply * Quick OOM, * Outside recommended range, * Long-lived pod with significant change. + * `EvictionRequirements` are all true. -`InPlaceOrRecreate` will attempt to apply a disruption-free update in place if it meets at least one +`InPlaceOrRecreate` will attempt to apply an update in place if it meets at least one of the following conditions: * Quick OOM, * Outside recommended range, -* Significant change. - -`InPlaceOrRecreate` when considering a disruption-free update in place ignore some conditions that -influence eviction decission in the `Recreate` mode: +* Long-lived Significant change. * [`CanEvict`] won't be checked and -* Pods with significant change can be updated even if they are not long-lived. - -`InPlaceOrRecreate` will attempt to apply updates that are **not** disruption-free in place under -the same conditions that apply to updates in the `Recreate` mode. +* [`EvictionRequirements`] won't be checked -`InPlaceOrRecreate` will attempt to apply updates that by eviction when: -* VPA already attempted to apply the update in-place and failed and -* it meets conditions for applying in the `Recreate` mode. [`CanEvict`]: https://github.com/kubernetes/autoscaler/blob/114a35961a85efdf3f36859350764e5e2c0c7013/vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go#LL100C10-L100C37 [by default less than 10 minutes]: https://github.com/kubernetes/autoscaler/blob/114a35961a85efdf3f36859350764e5e2c0c7013/vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go#L37 @@ -207,6 +203,7 @@ the same conditions that apply to updates in the `Recreate` mode. [by default 12h]: https://github.com/kubernetes/autoscaler/blob/114a35961a85efdf3f36859350764e5e2c0c7013/vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go#L35 [by default 10%]: https://github.com/kubernetes/autoscaler/blob/114a35961a85efdf3f36859350764e5e2c0c7013/vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go#L33 [Outside recommended range]: https://github.com/kubernetes/autoscaler/blob/114a35961a85efdf3f36859350764e5e2c0c7013/vertical-pod-autoscaler/pkg/updater/priority/priority_processor.go#L73 +[`EvictionRequirements`]: https://github.com/kubernetes/autoscaler/blob/54fe60ed4d4bb4cb89fe4abe11284d1bd6b06390/vertical-pod-autoscaler/pkg/updater/priority/scaling_direction_pod_eviction_admission.go ### Test Plan @@ -214,12 +211,12 @@ The following test scenarios will be added to e2e tests. The `InPlaceOrRecreate` tested in the following scenarios: * Admission controller applies recommendation to pod controlled by VPA. -* Disruption-free in-place update applied to all containers of a pod (request in recommendation bounds). -* Partial disruption-free update applied to some containers of a pod, some disruptive changes skipped (request in +* In-place update applied to all containers of a pod. +* Partial updates applied to some containers of a pod, some changes skipped (request in recommendation bounds). -* Disruptive in-place update applied to all containers of a pod (request out ouf recommendation bounds). -* Disruptive in-place update will fail. Pod should be evicted and the recommendation applied. -* Disruption free in-place updates fail, pod should not be evicted. +* In-place update will fail. Pod should be evicted and the recommendation applied. +* In-place update will fail but `CanEvict` is false, pod should not be evicted. +* In-place update will fail but `EvictionRequirements` are false, pod should not be evicted. ### Details still to consider @@ -231,4 +228,4 @@ Needs more research on how to scale down on memory safely. ## Implementation History - 2023-05-10: initial version -- 2025-02-07: Updates to align with latest changes to [KEP-1287](https://github.com/kubernetes/enhancements/issues/1287). +- 2025-02-19: Updates to align with latest changes to [KEP-1287](https://github.com/kubernetes/enhancements/issues/1287). From b62defb98b86ceb200a0c32d62d35c73b10051ba Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Wed, 19 Feb 2025 19:49:57 +0000 Subject: [PATCH 07/13] Small changes to adopt changes from KEP-1287 for 1.33 --- .../4016-in-place-updates-support/README.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index d8036749e11..5026fb3128a 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -94,9 +94,14 @@ two key features: Note that resize operations will NOT change the pod's quality of service (QoS) class. +Note that in the initial Beta version of in-place updates, [memory limit downscaling is forbidden] +for pods with `resizePolicy: PreferNoRestart`. This means that when VPA will attempt to apply the +patch, it will fail and VPA will need to fallback to a regular eviction (see below). + [In-place update of pod resources KEP]: https://github.com/kubernetes/enhancements/issues/1287 [`/resize` subresource]:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources#api-changes [`ResizePolicy`]: https://github.com/kubernetes/api/blob/4dccc5e86b957cea946a63c4f052ee7dec3946ce/core/v1/types.go#L2636 +[memory limit downscaling is forbidden]: https://github.com/kubernetes/enhancements/pull/5089 ## Design Details @@ -153,13 +158,15 @@ The VPA updater will evict a pod to actuate a recommendation if it attempted to recommendation in place and failed. VPA updater will consider that the update failed if: -* The pod has `.status.resize: Infeasible` or -* The pod has `.status.resize: Deferred` and more than 5 minutes elapsed since the update or -* The pod has `.status.resize: InProgress` and more than 1 hour elapsed since the update: - * There seems to be a bug where containers that say they need to be restarted get stuck in update, hopefully it gets - fixed and we don't have to worry about this by beta. +* The pod has condition `PodResizePending` with reason `Infeasible` or +* The pod has condition `PodResizePending` with reason `Deferred` and more than 5 minutes elapsed + since the update or +* The pod has condition `PodResizing` and more than 1 hour elapsed since the update or * Patch attempt returns an error. +Note that in the initial version of In-Place updates, memory limit downscaling will always fail. +This means VPA will need to evict the pod normally for this change to happen. + ### Comparison of `UpdateMode`s Today, VPA updater considers the following conditions when deciding if it should apply an update: From c34f019bc0e08ee591470705f23c56329149dde8 Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Wed, 19 Feb 2025 19:59:11 +0000 Subject: [PATCH 08/13] Add note about ResizePolicy and VPA. --- .../4016-in-place-updates-support/README.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 5026fb3128a..af4f649299c 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -164,8 +164,14 @@ VPA updater will consider that the update failed if: * The pod has condition `PodResizing` and more than 1 hour elapsed since the update or * Patch attempt returns an error. -Note that in the initial version of In-Place updates, memory limit downscaling will always fail. -This means VPA will need to evict the pod normally for this change to happen. +Note that in the initial version of In-Place updates, memory limit downscaling will always fail +the patch operation. This means VPA will need to evict the pod normally for this change to happen. + +#### A note on `ResizePolicy`. + +VPA does not care and should not care about a container's `ResizePolicy` setting. In the new mode, +it will simply issue the `/resize` request and let the underlying machinery apply the resize +operation in a way that complies with the user's specification. ### Comparison of `UpdateMode`s From 1969f47f62175bf97a20e025fc9bd5202157c31d Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Wed, 19 Feb 2025 20:11:12 +0000 Subject: [PATCH 09/13] Edits to address comments from sftim --- .../4016-in-place-updates-support/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index af4f649299c..809c5f7f257 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -63,8 +63,8 @@ Add a new supported value of [`UpdateMode`]: * `InPlaceOrRecreate` -Here we specify `OrRecreate` to make sure the user explicitly knows that the pod may be -restarted. +Here we specify `InPlaceOrRecreate` to make sure the user explicitly knows that the existing pod +may be replaced. [`UpdateMode`]: https://github.com/kubernetes/autoscaler/blob/71b489f5aec3899157b37472cdf36a1de223d011/vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go#L124 @@ -105,14 +105,14 @@ patch, it will fail and VPA will need to fallback to a regular eviction (see bel ## Design Details -Today, only the VPA admission controller is responsible for changing the pod spec. +Prior to this AEP, only the VPA admission controller was responsible for changing the pod spec. The VPA updater is responsible for evicting pods so that the admission controller can change them during admission. In the newly added `InPlaceOrRecreate` mode, the VPA Updater will attempt to execute in-place -updates FIRST. If it is unable to process an in-place update in time, it will evict the pod to force -a change. +updates _FIRST_. If it is unable to process an in-place update in time, it will evict the pod to +force a change. This will effectively match the current behavior in `Auto` except that resizes will first be attempted in-place. @@ -136,9 +136,9 @@ starting pods just as it does for VPAs in `Initial`, `Auto`, and `Recreate` mode ### 2. In-Place Updates (**NEW**) {#in-place} -In the `InPlaceOrRecreate` modes VPA updater will attempt to apply updates that require container -restart in-place. It will update them under the same conditions that would trigger update with -`Recreate` mode. That is it will apply an in-place update if: +In the `InPlaceOrRecreate` modes, and for updates that require a container restart, the VPA updater +will attempt to apply updates in place. It will update them under the same conditions that would +trigger an update with `Recreate` mode. That is it will apply an in-place update if: * Any container has a request below the corresponding `LowerBound` or * Any container has a request above the corresponding `UpperBound` or From 4ea61cbc7a31a4dd63498c84398a8491d7d2c93c Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Wed, 19 Feb 2025 20:13:50 +0000 Subject: [PATCH 10/13] fix typo --- .../enhancements/4016-in-place-updates-support/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 809c5f7f257..e0df7bd33db 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -122,7 +122,7 @@ In the future, this logic may be improved to: * Make changes that are only attempted using in-place resizes and wouldn't ultimately result in an eviction on failure. * In the case of failure, make smaller updates to circumvent a node that does not have enough - headroom to accept the full resize but could accomodate a smaller one. + headroom to accept the full resize but could accommodate a smaller one. We classify two types of updates in the context of this new mode: From 6214833a805572dfbdafd426435882bf69134c6a Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Thu, 20 Feb 2025 20:02:33 +0000 Subject: [PATCH 11/13] URL and formatting updates --- .../4016-in-place-updates-support/README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index e0df7bd33db..371c3da5fb0 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -2,13 +2,14 @@ - [Summary](#summary) + - [A Note On Disruptions](#a-note-on-disruptions) - [Goals](#goals) - [Non-Goals](#non-goals) - [Proposal](#proposal) - [Context](#context) - [Design Details](#design-details) - - [1. Applying Updates During Pod Admission](#pod-admission) - - [2. In-Place Updates (**NEW**)](#in-place) + - [1. Applying Updates During Pod Admission](#applying-updates-during-pod-admission) + - [2. In-Place Updates](#in-place-updates) - [Comparison of `UpdateMode`s](#comparison-of-updatemodes) - [Test Plan](#test-plan) - [Implementation History](#implementation-history) @@ -30,9 +31,9 @@ introduced as separate enhancement proposals. [in-place update feature]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources [alpha feature since 1.27]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.27.md#api-change-3 -[beta in 1.33]: https://github.com/orgs/kubernetes/projects/178/views/1 +[beta in 1.33]: https://github.com/kubernetes/enhancements/blob/4c3449517e1a7cf00907fd8a6dc780350275035a/keps/sig-node/1287-in-place-update-pod-resources/kep.yaml#L38-L41 -### A Note On Disruptions {#disruptions} +### A Note On Disruptions It is important to note that **VPA cannot guarantee NO disruptions**. This is because the underlying container runtime is responsible for actuating the resize operation and there are no @@ -101,7 +102,7 @@ patch, it will fail and VPA will need to fallback to a regular eviction (see bel [In-place update of pod resources KEP]: https://github.com/kubernetes/enhancements/issues/1287 [`/resize` subresource]:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/1287-in-place-update-pod-resources#api-changes [`ResizePolicy`]: https://github.com/kubernetes/api/blob/4dccc5e86b957cea946a63c4f052ee7dec3946ce/core/v1/types.go#L2636 -[memory limit downscaling is forbidden]: https://github.com/kubernetes/enhancements/pull/5089 +[memory limit downscaling is forbidden]: https://github.com/kubernetes/enhancements/blob/4c3449517e1a7cf00907fd8a6dc780350275035a/keps/sig-node/1287-in-place-update-pod-resources/README.md#memory-limit-decreases ## Design Details @@ -129,12 +130,12 @@ We classify two types of updates in the context of this new mode: 1. Updates on pod admission 2. In-place updates -### 1. Applying Updates During Pod Admission {#pod-admission} +### 1. Applying Updates During Pod Admission For VPAs using the new `InPlaceOrRecreate` mode, the VPA Admission Controller will apply updates to starting pods just as it does for VPAs in `Initial`, `Auto`, and `Recreate` modes. -### 2. In-Place Updates (**NEW**) {#in-place} +### 2. In-Place Updates In the `InPlaceOrRecreate` modes, and for updates that require a container restart, the VPA updater will attempt to apply updates in place. It will update them under the same conditions that would From 047e6ad5c8e8954195fc78881d076a3540cbe31c Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Thu, 20 Feb 2025 20:03:56 +0000 Subject: [PATCH 12/13] more anchor fixes --- .../enhancements/4016-in-place-updates-support/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 371c3da5fb0..4b7e3d1e977 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -8,8 +8,8 @@ - [Proposal](#proposal) - [Context](#context) - [Design Details](#design-details) - - [1. Applying Updates During Pod Admission](#applying-updates-during-pod-admission) - - [2. In-Place Updates](#in-place-updates) + - [Applying Updates During Pod Admission](#applying-updates-during-pod-admission) + - [In-Place Updates](#in-place-updates) - [Comparison of `UpdateMode`s](#comparison-of-updatemodes) - [Test Plan](#test-plan) - [Implementation History](#implementation-history) @@ -130,12 +130,12 @@ We classify two types of updates in the context of this new mode: 1. Updates on pod admission 2. In-place updates -### 1. Applying Updates During Pod Admission +### Applying Updates During Pod Admission For VPAs using the new `InPlaceOrRecreate` mode, the VPA Admission Controller will apply updates to starting pods just as it does for VPAs in `Initial`, `Auto`, and `Recreate` modes. -### 2. In-Place Updates +### In-Place Updates In the `InPlaceOrRecreate` modes, and for updates that require a container restart, the VPA updater will attempt to apply updates in place. It will update them under the same conditions that would From a4d2d6c6e8f6b378fef0fb60a72c28bec34d611a Mon Sep 17 00:00:00 2001 From: Ray Wainman Date: Thu, 20 Feb 2025 20:53:41 +0000 Subject: [PATCH 13/13] added clarification on eviction after in-place failure --- .../enhancements/4016-in-place-updates-support/README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md index 4b7e3d1e977..aa0c23536a0 100644 --- a/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md +++ b/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support/README.md @@ -153,10 +153,13 @@ be able to perform an eviction, including: * If `CanEvict` is false. * If any of the `EvictionRequirements` on the VPA are not true. -These additional resizes can be attempted because the eviction fallback would fail anyway. +If the in-place resize operation fails in this case, VPA can still proceed with the normal eviction +path which would get blocked anyway due to these conditions preventing it from happening. The VPA updater will evict a pod to actuate a recommendation if it attempted to apply the -recommendation in place and failed. +recommendation in place and failed. This will happen even if we attempted the in-place resize +for conditions that normally would not lead to an eviction. This is safe because the eviction would +be prevented anyway. VPA updater will consider that the update failed if: * The pod has condition `PodResizePending` with reason `Infeasible` or