Skip to content

feat(rescheduling): gpuFragmentation eligible by default + env kill switch - #13

Merged
pfernandes21 merged 3 commits into
exa/masterfrom
devin/1786725018-repack-opt-out
Aug 14, 2026
Merged

feat(rescheduling): gpuFragmentation eligible by default + env kill switch#13
pfernandes21 merged 3 commits into
exa/masterfrom
devin/1786725018-repack-opt-out

Conversation

@pfernandes21

@pfernandes21 pfernandes21 commented Aug 14, 2026

Copy link
Copy Markdown

What type of PR is this?

/kind feature

What this PR does / why we need it:

Two changes to the gpuFragmentation repack strategy:

  1. Eligibility flips from opt-in to opt-out. Previously a pod was only movable if it carried exa.ai/repack-eligible=true; now every pod is eligible by default, and setting exa.ai/repack-eligible=false excludes it:
-	EligibleLabel   string `mapstructure:"eligibleLabel"`   // require label == "true"
+	OptOutLabel     string `mapstructure:"optOutLabel"`     // skip when label == "false"

-	if sole.Pod.Labels[conf.EligibleLabel] != "true" {
+	if sole.Pod.Labels[conf.OptOutLabel] == "false" {
 		return nil
 	}
  1. Env-var kill switch. Setting EXA_GPU_REPACK_DISABLED=true on the scheduler process makes the strategy return no victims, independent of the scheduler configmap:
kubectl -n volcano set env deploy/<scheduler> EXA_GPU_REPACK_DISABLED=true   # off (pod restarts)
kubectl -n volcano set env deploy/<scheduler> EXA_GPU_REPACK_DISABLED-      # back on

All other safety gates are unchanged and still bound the blast radius: only negative-priority pods (maxVictimPriority: -1 default — priority ≥ 0 is never touched), sole GPU pod on its node, controller-owned, single-member PodGroup, no karpenter.sh/do-not-disrupt, strictly fuller feasible destination, pool cooldown, per-PodGroup eviction cap, and one victim per pool per pass.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Behavioral widening: after rollout, any negative-priority sole-GPU pod becomes repackable without labeling. Workloads that must never be repacked should set exa.ai/repack-eligible=false (or a priority class ≥ 0, or karpenter.sh/do-not-disrupt). Unit tests updated: unlabeled pod is now selected; =false is skipped; 14/14 pass.

Does this PR introduce a user-facing change?

gpuFragmentation repacking is now opt-out (exa.ai/repack-eligible=false) instead of opt-in, and can be disabled entirely with EXA_GPU_REPACK_DISABLED=true on the scheduler.

Link to Devin session: https://app.devin.ai/sessions/6351ff07ed074c55a521b26d6af34c1b
Requested by: @pfernandes21

@pfernandes21 pfernandes21 self-assigned this Aug 14, 2026
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot changed the title feat(rescheduling): gpuFragmentation eligible by default — remove opt-in label requirement feat(rescheduling): gpuFragmentation eligible by default + env kill switch Aug 14, 2026
@pfernandes21
pfernandes21 merged commit 0051601 into exa/master Aug 14, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant