feat(scheduler): evict for deferred in-place resizes behind opt-in action - #2051
Draft
gshaibi wants to merge 36 commits into
Draft
feat(scheduler): evict for deferred in-place resizes behind opt-in action#2051gshaibi wants to merge 36 commits into
gshaibi wants to merge 36 commits into
Conversation
Adds design document for effective-request accounting and best-effort webhook admission for in-place pod resizes. Fixes #1906. Signed-off-by: gshaibi <gshaibi@nvidia.com>
…nd queue admission Implements the design from PR #1997: 1. Effective-request accounting (KEP-1287): - Preserve ContainerStatus.Resources and AllocatedResources in the scheduler informer compact transform (pod_transform.go) so resize state is visible to the scheduler. - Compute effective pod requests as max(spec, enacted, allocated) per container; exclude spec when PodResizePending=Infeasible so infeasible resize targets do not inflate accounting. 2. Best-effort pods/resize admission webhook: - New PodResizeValidator validates pods/resize subresource updates. - Computes delta = max(proposed - old spec, 0) and rejects if any queue on the hierarchy would exceed its CPU/memory limit (all workloads) or CPU/memory quota (non-preemptible workloads). - failurePolicy=ignore so webhook unavailability never blocks a resize. Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
- operator: add pods/resize webhook entry (failurePolicy=Ignore) to validatingWCForKAIConfig so the handler is reachable in production - admission: register scheduling/v2 (Queue) and scheduling/v2alpha2 (PodGroup) in the manager scheme; add RBAC markers for PodGroups, Queues, and PriorityClasses so the client can fetch them at runtime - scheduler: guard isPodResizeInfeasible with Status==ConditionTrue to avoid acting on a stale or transitional condition - admission: replace specPodRequests delta with per-resource, per-container delta using max(enacted, allocated) as the effective-old baseline; skip resources unchanged in this resize so a stale infeasible spec target does not produce a spurious delta - admission: treat quota==0 as a finite boundary (use >= 0, matching the limit check) so non-preemptible upsizes are blocked when quota is zero - scheduler: apply the KEP-1287 effective-request model to restartable init containers (native sidecars) in initContainerEffects so a sidecar downsize in progress is not undercounted Signed-off-by: gshaibi <gshaibi@nvidia.com>
…pod resize validator - Add InPlacePodResize config struct to Admission API with ValidateQuota and BlockUpsizeOnBoundedQueues fields (defaults: true / false) - Add --validate-pod-resize-quota and --block-upsize-on-bounded-queues CLI flags - Wire flags from operator buildArgsList through to PodResizeValidator constructor - validateQuota=false: early-return allows all resizes without quota checks - blockUpsizeOnBoundedQueues=true: reject any upsize on queues with finite limit (all workloads) or finite quota (non-preemptible), regardless of allocation - Add tests covering both new modes Signed-off-by: gshaibi <gshaibi@nvidia.com>
…ake validate - Fix import ordering in app.go (goimports sort) - Fix struct field alignment in options.go (gofmt) - Remove trailing blank line in effective_requests.go - Regenerate CRD with inPlacePodResize fields - Regenerate admission RBAC with queues/podgroups/priorityclasses rules Signed-off-by: gshaibi <gshaibi@nvidia.com>
…ine, and redistribution Three bugs in podResizeDelta addressed: - Sidecar upsizes bypassed admission: now iterates restartable init containers (RestartPolicy=Always) using InitContainerStatuses, matching the scheduler's initContainerEffects accounting. - Non-infeasible old spec was excluded from baseline: for normal/Deferred/InProgress pods the queue accounts for max(spec, enacted, allocated); the webhook now mirrors this instead of always using only max(enacted, allocated). - Per-container aggregation counted redistribution as growth: moving CPU from container A to container B produced a positive delta. Switched to pod-level aggregation (sum across all containers before diffing). The new structure: accumulate newSpecSum, oldSpecSum, effectiveOldSum at the pod level; skip a resource if newSpecSum[r] == oldSpecSum[r] (unchanged by this resize); delta = max(0, newSpecSum[r] - effectiveOldSum[r]). Add three tests: sidecar denied, non-infeasible baseline, redistribution allowed. Signed-off-by: gshaibi <gshaibi@nvidia.com>
…entation-2f767b # Conflicts: # pkg/operator/operands/admission/resources.go # pkg/scheduler/cache/pod_transform.go Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
…helpers Replaces hand-rolled KEP-1287 aggregation with upstream resource.AggregateContainerRequests(UseStatusResources: true). - delete effective_requests.go entirely (3 funcs -> upstream) - collapse getPodResourceWithoutInitContainers + initContainerEffects into a single upstream call - webhook uses resource.IsPodResizeInfeasible instead of a local copy - retarget effective-request tests at the public aggregation path KNOWN FAILURE: upstream IsPodResizeInfeasible does not check condition.Status == True, so a stale Infeasible condition with Status=False is treated as infeasible. Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
…ion test Upstream keys off Reason alone and ignores condition.Status, so the Status==True guard added in #2021 does not survive the move to k8s.io/component-helpers. Accepts upstream semantics and pins the difference so a future upstream tightening surfaces as a test failure rather than a silent change. Signed-off-by: gshaibi <gshaibi@nvidia.com>
…helpers-prototype refactor(scheduler): delegate effective-request accounting to k8s.io/component-helpers
…entation-2f767b Signed-off-by: gshaibi <gshaibi@nvidia.com>
…unting_test.go The effective_requests.go source it was named after was removed when aggregation moved to k8s.io/component-helpers. The tests now cover KAI's wiring of the upstream helper through getPodResourceRequest. Signed-off-by: gshaibi <gshaibi@nvidia.com>
…cePodVerticalScaling Signed-off-by: gshaibi <gshaibi@nvidia.com>
…g hierarchy - compactResizeConditions -> compactConditions, matching sibling naming (functions are named for the field they compact); drop stale 'generation-aware' wording - document that BlockUpsizeOnBoundedQueues has no effect when ValidateQuota is false, in API comments, CLI flag help, and CRD Signed-off-by: gshaibi <gshaibi@nvidia.com>
…cated resources Queue.Status.Allocated is derived from PodGroup.Status.ResourcesStatus.Allocated, which summed raw pod spec requests. An Infeasible resize target therefore inflated queue allocated (e.g. a rejected 64-CPU resize charged 64 CPU), diverging from scheduler-internal accounting and double-counting in the resize webhook's limit check. Found by live-cluster testing. Switch to the same upstream aggregation the scheduler uses (AggregateContainerRequests): KEP-1287 effective requests for allocated (UseStatusResources), spec-only for requested. This also closes a pre-existing undercount: init-phase peak and sidecars now count toward podgroup/queue allocated, matching what the scheduler has always reserved internally. Signed-off-by: gshaibi <gshaibi@nvidia.com>
…tion, dedup capacity checks Move IsPreemptible (with its PriorityClass fallback chain: specific -> global default -> DefaultPodGroupPriority) from podgroupcontroller utilities to pkg/common/podgroup, and use it in the resize webhook instead of a divergent local resolver. The webhook assumed priority 0 for unset/missing priority classes while the controller fell back to defaults - so a pod counted in AllocatedNonPreemptible could skip the webhook's quota check entirely. The checker and the accountant now share one implementation. Merge checkLimit and checkNonPreemptibleQuota into checkCapacityBound, parameterized by bound and allocated pool. The earlier quota>0 vs limit>=0 sentinel bug was drift between these two copies. Signed-off-by: gshaibi <gshaibi@nvidia.com>
E2E suite (test/e2e/suites/resize/) covers the resize admission webhook (deny past queue limit, allow within, downsizes) and effective-request accounting end to end (infeasible resize target must not inflate Queue.status.allocated). Skips on clusters older than 1.33. Resize calls retry on conflict: the kubelet writes pod status concurrently during a resize. User docs (docs/in-place-resize/) cover the effective-request charging model, webhook semantics including best-effort fail-open behavior, and the inPlacePodResize config fields. Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
GetConnectivity stores the calling node's context; ginkgo cancels it when the node exits, so a TestContext created in BeforeEach fails every client call from the It body with 'context canceled'. Follow the quota-suite pattern: connectivity, skips, and cleanup live inside each spec. Signed-off-by: gshaibi <gshaibi@nvidia.com>
Replace the raw admission.Handler with the generic admission.Validator[*corev1.Pod] and register through the webhook builder with WithValidatorCustomPath, matching the sibling webhooks. The custom path is still required: the canonical /validate--v1-pod path belongs to the general pod validator. Removes the manual decoder, the scheme constructor parameter, and the request-encoding helpers in tests, which now call ValidateUpdate directly. Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
…t guarantee Signed-off-by: gshaibi <gshaibi@nvidia.com>
…resizes Signed-off-by: gshaibi <gshaibi@nvidia.com>
Remove comments that restate the code (capacity-check section headers, inline continue/return annotations) and compress multi-line rationales to their load-bearing sentences. Signed-off-by: gshaibi <gshaibi@nvidia.com>
…nt charge Found during manual testing: a follow-up upsize issued before the kubelet enacts a pending downsize is compared against the transient (still-charged) value and can settle above the queue limit. Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
Signed-off-by: gshaibi <gshaibi@nvidia.com>
Detect pods whose in-place resize the kubelet marked Deferred and evict victims on their node so the resize can be enacted. Victim eligibility mirrors a new pod of that size: same-queue preempt rules by priority, cross-queue reclaim rules gated on fair share, validated by the same scenario validators. All-or-nothing per resizing pod. The action is registered but not part of the default actions list. Part of #1872 Signed-off-by: gshaibi <gshaibi@nvidia.com>
Default priority 150 (after preempt); enabled per shard via spec.actions.resizeeviction.enabled. Part of #1872 Signed-off-by: gshaibi <gshaibi@nvidia.com>
Part of #1872 Signed-off-by: gshaibi <gshaibi@nvidia.com>
Part of #1872 Signed-off-by: gshaibi <gshaibi@nvidia.com>
gshaibi
added a commit
that referenced
this pull request
Aug 10, 2026
…erred charge as a contract Generation-aware Infeasible was rejected during implementation (the observedGeneration field requires a non-GA gate; upstream reason-only semantics adopted and pinned by a characterization test) - update the design to match. Note that the Deferred max(spec, actual) charge is a dependency of deferred-resize eviction (#1872 / #2051): the reserved target prevents backfill thrash after victim eviction. Signed-off-by: gshaibi <gshaibi@nvidia.com>
📊 Performance Benchmark ResultsComparing PR (
|
Merged
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements deferred resize eviction — goal 3 of the in-place pod resize design, left out of scope there.
When the kubelet marks an in-place resize
Deferred(PodResizePendingcondition) because the node lacks capacity, KAI currently frees nothing, so the resize can starve indefinitely even when the node runs workloads a new pod of that size would be allowed to displace.This PR adds a new scheduler action,
resizeeviction, disabled by default, that evicts victims on the resizing pod's node until the kubelet can enact the resize:PodResizePending/Deferredfor the current resize generation; the still-unallocated delta isspec − actualper resizable resource (CPU/memory).max(spec, actual)on both node and queue, so the demand is a node-local scalar shortfall (Used − Releasing − other deferred deltas − Allocatable), and the freed capacity stays reserved across cycles — that is also the thrash guard. No virtual pending task and no scenario-solver surgery; the pod is bound, nothing needs placement.PreemptVictimFilter), cross-queue victims under reclaim rules (CanReclaimResources+ReclaimVictimFilter), ordered by the standard victims queue, gang/elastic semantics viaGetTasksToEvict, and the accumulated victim set validated byPreemptScenarioValidator/ReclaimScenarioValidatorFnbefore commit. Eviction is all-or-nothing per resizing pod.spec.actions.resizeeviction.enabled: true(default priority 150, after preempt); raw-config users appendresizeevictionto the actions string.Design doc:
docs/developer/designs/in-place-pod-resize/deferred-resize-eviction.md(rationale, decisions, known limitations). User docs updated indocs/in-place-resize/README.mdanddocs/operator/scheduler-config-customization.md.Related Issues
Fixes #1872
Checklist
make changelog(or applied theskip-changeloglabel). Do not editCHANGELOG.mddirectly — pending fragments are folded into it at release time.Breaking Changes
None. The action is off by default; behavior is unchanged unless explicitly enabled.
Additional Notes
pkg/scheduler/actions/resizeeviction/resizeeviction_test.go(preempt/reclaim eligibility, all-or-nothing, gang victims, multi-deferred shortfall exclusion, already-enactable no-op) andpkg/scheduler/api/pod_info/resize_test.go(condition generation-awareness, delta computation).test/e2e/suites/resize/deferred_eviction_test.goenables the action on the default shard, drives a realpods/resizeintoDeferred, and asserts the victim is evicted and the resize enacts.