Skip to content

feat(rescheduling): optional crossPool mode for gpuFragmentation - #18

Merged
pfernandes21 merged 1 commit into
exa/masterfrom
devin/1788176109-repack-cross-pool
Aug 31, 2026
Merged

feat(rescheduling): optional crossPool mode for gpuFragmentation#18
pfernandes21 merged 1 commit into
exa/masterfrom
devin/1788176109-repack-cross-pool

Conversation

@pfernandes21

Copy link
Copy Markdown

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds an opt-in crossPool flag (default false) to the gpuFragmentation rescheduling strategy. Today candidate sources/destinations are grouped per karpenter.sh/nodepool, so consolidation never crosses a pool boundary and unlabeled (e.g. on-prem) nodes are invisible.

With crossPool: true:

  • every GPU node in the cluster forms one candidate set, including nodes without the pool label;
  • the workload's own node selection — nodeSelector, required affinity, taints/tolerations — evaluated through the existing session predicate path, is the only placement filter;
  • the cooldown clock, one-drain-per-pass budget, and metrics pool label become cluster-wide (label value cross-pool).

Planner change is confined to the grouping loop:

if conf.CrossPool {
    pools[crossPoolName] = append(pools[crossPoolName], node)
    continue
}
pool, ok := node.Node.Labels[conf.PoolLabel]
...

Everything downstream (FFD simulation over cloned idle capacity, fractional-fullness destination ordering, atomic move sets, priority ceiling/tie-break, per-PodGroup eviction caps, stamping) is unchanged. Default behavior with crossPool: false is byte-for-byte the existing per-pool behavior.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

  • New tests: cross-pool drain that per-pool mode blocks, unlabeled-node destination, and a predicate veto blocking an otherwise-fitting cross-pool move; all existing per-pool tests unchanged and passing.
  • Predicates are the only guard on placement: pods with loose/no node selection may move across instance types, capacity types, or AZs. That is the explicitly chosen semantic.
  • go test ./pkg/scheduler/plugins/rescheduling/... passes; go vet/gofmt clean.

Does this PR introduce a user-facing change?

gpuFragmentation gains an opt-in crossPool option: when true, GPU repacking considers all GPU nodes cluster-wide (including nodes without the pool label) and relies solely on workload node selection via scheduler predicates for placement; cooldown and drain budget apply cluster-wide. Default remains per-pool.

Link to Devin session: https://app.devin.ai/sessions/9f5f310b988c4566869471b54b1154dd
Open in Devin Desktop: https://app.devin.ai/desktop/session/9f5f310b988c4566869471b54b1154dd?variant=devin
Requested by: @pfernandes21

With crossPool enabled every GPU node forms one candidate set: the
workload's own node selection (nodeSelector, required affinity, taints),
evaluated through the session predicates, is the only placement filter.
Unlabeled nodes are included, and the cooldown clock and
one-drain-per-pass budget apply cluster-wide. Default remains per-pool.

Assisted-by: devin:claude
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration
devin-ai-integration Bot changed the base branch from master to exa/master August 31, 2026 11:39
@pfernandes21
pfernandes21 merged commit c7a0e27 into exa/master Aug 31, 2026
8 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant