Skip to content

Conversation

@sohankunkerkar
Copy link
Member

@sohankunkerkar sohankunkerkar commented Jan 7, 2026

When the scheduler admits a new workload slice but fails to finish the old slice, the job controller should clean it up. This fixes a race condition where old workload slices weren't being cleaned up during scale-up events,
causing the RayJob InTreeAutoscaling e2e test to see 4 workloads instead of 3.

What type of PR is this?

/kind bug
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #8447

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Fix EnsureWorkloadSlices to finish old slice when new is admitted as replacement

Copilot AI review requested due to automatic review settings January 7, 2026 21:04
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. kind/flake Categorizes issue or PR as related to a flaky test. labels Jan 7, 2026
@netlify
Copy link

netlify bot commented Jan 7, 2026

Deploy Preview for kubernetes-sigs-kueue canceled.

Name Link
🔨 Latest commit 3d31046
🔍 Latest deploy log https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/695f49aa5df3670008759617

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 7, 2026
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sohankunkerkar
Once this PR has been reviewed and has the lgtm label, please assign gabesaba, pbundyra for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jan 7, 2026
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug in the workload slice cleanup logic during elastic job scale-up operations. When the scheduler admits a new workload slice and sets the WorkloadSliceReplacementFor annotation, the old slice should be finished. Previously, the code only finished the old slice if it lost quota reservation or was evicted, missing the replacement annotation scenario.

Key Changes

  • Added a third condition to finish old workload slices when the new slice has the replacement annotation pointing to it
  • Added comprehensive inline documentation explaining the three conditions for finishing old slices
  • Added test coverage for the replacement annotation scenario

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pkg/workloadslicing/workloadslicing.go Enhanced EnsureWorkloadSlices case 2 logic to check for replacement annotation and finish old slices accordingly
pkg/workloadslicing/workloadslicing_test.go Added new test case "TwoWorkloads_OldWithReservedQuota_NewWithReplacementAnnotation" to verify the fix

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@sohankunkerkar sohankunkerkar changed the title Fix workload slice cleanup when replacement annotation exists [WIP] Fix workload slice cleanup when replacement annotation exists Jan 7, 2026
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 7, 2026
…replacement

When the scheduler admits a new workload slice but fails to finish the old
slice, the job controller should clean it up. This fixes a race condition
where old workload slices weren't being cleaned up during scale-up events,
causing the RayJob InTreeAutoscaling e2e test to see 4 workloads instead of 3.

Signed-off-by: Sohan Kunkerkar <[email protected]>
@sohankunkerkar sohankunkerkar changed the title [WIP] Fix workload slice cleanup when replacement annotation exists [WIP] Fix EnsureWorkloadSlices to finish old slice when new is admitted as replacement Jan 8, 2026
@sohankunkerkar sohankunkerkar changed the title [WIP] Fix EnsureWorkloadSlices to finish old slice when new is admitted as replacement Fix EnsureWorkloadSlices to finish old slice when new is admitted as replacement Jan 8, 2026
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. kind/flake Categorizes issue or PR as related to a flaky test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[flaky test] Kuberay Should run a rayjob with InTreeAutoscaling

2 participants