Skip to content

fix: prefer accelerators in heterogeneous placement - #2252

Open
abendrothj wants to merge 1 commit into
exo-explore:mainfrom
abendrothj:fix/2180-accelerator-aware-placement
Open

fix: prefer accelerators in heterogeneous placement#2252
abendrothj wants to merge 1 commit into
exo-explore:mainfrom
abendrothj:fix/2180-accelerator-aware-placement

Conversation

@abendrothj

Copy link
Copy Markdown

Motivation

Backend compatibility alone can leave a CPU-only singleton tied with an accelerator singleton. The existing free-memory tiebreaker can then assign the full model to the CPU while CUDA sits idle.

Changes

  • score valid candidate cycles by accelerator-backed nodes
  • rank Metal/CUDA capability ahead of download locality and free-memory tiebreakers
  • preserve existing smallest-cycle, required-node, backend, and RDMA filtering
  • add a regression test where a lower-memory CUDA node beats a higher-memory CPU node

Testing

  • uv run pytest src/exo/master/tests/test_placement.py -k "accelerator or backend"
  • uv run basedpyright src/exo/master/placement.py src/exo/master/tests/test_placement.py
  • uv run ruff check src/exo/master/placement.py src/exo/master/tests/test_placement.py

Fixes #2180


Refiled from #2213, which was closed unintentionally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Heterogeneous cluster scheduling may assign full inference to CPU worker despite available CUDA GPU

1 participant