Skip to content

feat(rescheduling): gpuFragmentation strategy for single-pod GPU repacking - #10

Merged
pfernandes21 merged 1 commit into
exa/masterfrom
devin/1786541784-gpu-fragmentation-rescheduling
Aug 12, 2026
Merged

feat(rescheduling): gpuFragmentation strategy for single-pod GPU repacking#10
pfernandes21 merged 1 commit into
exa/masterfrom
devin/1786541784-gpu-fragmentation-rescheduling

Conversation

@pfernandes21

Copy link
Copy Markdown

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds a gpuFragmentation strategy to the rescheduling plugin, evicted via the existing shuffle action, to reclaim GPU fragmentation on delphi (e.g. 3/8 + 1/8 nodes that never re-pack after neighbours finish). Volcano binpacks only at arrival; nothing re-optimizes running pods. This strategy deschedules — the pod's controller recreates the replacement and normal allocate + binpack scoring places it on the fuller node. Nothing is pinned or bound by the strategy.

Per interval (rescheduling plugin's existing interval plumbing), per node pool (karpenter.sh/nodepool), it selects at most one victim:

victim ⇐ node where
  0 < gpuUsed < gpuAllocatable
  ∧ the node's ONLY GPU task is Running, opted in (exa.ai/repack-eligible=true),
    not karpenter.sh/do-not-disrupt, controller-owned,
    and its PodGroup has exactly 1 member (minMember ≤ 1)
  ∧ pool cooldown clock (node ann. exa.ai/repack-last-eviction) expired
    — missing = allowed; unparseable/future = fail closed, holds the pool
  ∧ PodGroup eviction cap (ann. exa.ai/repack-evictions) unspent
  ∧ ∃ strictly fuller node in the pool where the task fits:
      Idle resources ∧ ssn.PrePredicateFn/PredicateFn on an unbound
      DeepCopy of the pod (NodeName cleared, so nodeaffinity doesn't
      trivially reject every node but the source)

dryRun: true by default — logs the would-be move and evicts nothing. In live mode, the cooldown clock and PodGroup eviction count are patched durably before returning the victim; a failed stamp aborts the move, so budgets can never be overspent across scheduler restarts.

This replaces the controller-side repacker in exa-labs/monorepo#125619: doing it in-session reuses the real predicates/scoring instead of reimplementing them, at the cost of running inside vc-scheduler.

Enabling on delphi needs a scheduler config change (separate monorepo PR):

actions: "enqueue, allocate, preempt, backfill, reclaim, shuffle"
tiers:
- plugins:
  - name: rescheduling
    enableVictim: true
    arguments:
      interval: 5m
      strategies:
      - name: gpuFragmentation
        params: { dryRun: true }

Which issue(s) this PR fixes:

NONE

Special notes for your reviewer:

  • shuffle evicts whatever VictimTasks returns; the strategy therefore caps itself (maxVictims: 1 cluster-wide by default) and one victim per pool.
  • The destination proof shows a fit exists today; the replacement is not reserved there. If it lands back on the source, cooldown + the per-PodGroup cap bound the cost to one wasted restart — no thrash loop.
  • Live mode additionally needs vc-scheduler RBAC for nodes patch and podgroups patch; the strategy fails closed (skips the move) without it.
  • 12 unit tests cover selection, opt-in/protection/gang gates, cooldown (incl. unparseable annotation), eviction cap, destination fit/predicate veto, and probe unbinding.

Does this PR introduce a user-facing change?

Added a `gpuFragmentation` rescheduling strategy (dry-run by default) that deschedules a single opted-in GPU pod per node pool when its node would empty of GPU work and a fuller node in the pool can host it.

Link to Devin session: https://app.devin.ai/sessions/6351ff07ed074c55a521b26d6af34c1b
Requested by: @pfernandes21

…repacking

Evicts at most one opted-in GPU pod per node pool whose departure empties
its node of GPU work, after proving a strictly fuller node in the same pool
passes resources and session predicates. The replacement is recreated by
the pod's controller and scheduled normally; binpack scoring steers it to
the fuller node. Dry-run by default; cooldown and per-PodGroup eviction
caps are the anti-thrash mechanism, recorded durably on cluster objects
before eviction.
@pfernandes21 pfernandes21 self-assigned this Aug 12, 2026
@devin-ai-integration

Copy link
Copy Markdown

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@pfernandes21
pfernandes21 merged commit 18f1051 into exa/master Aug 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant