Add GPU benchmarking infrastructure with DiffEqGPU proof-of-concept #1440
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds GPU benchmarking support to SciMLBenchmarks.jl: CI Configuration: - Add run_gpu_benchmark.yml template for GPU jobs using juliagpu queue - Modify launch_benchmarks.yml to separate CPU and GPU benchmark triggers - Update build_benchmark.sh with GPU environment setup and CUDA verification Proof-of-Concept Benchmark: - Add benchmarks/GPU/ directory with DiffEqGPU Lorenz ensemble benchmark - Compare CPU (EnsembleThreads) vs GPU (EnsembleGPUKernel) performance - Benchmark across 100 to 100,000 trajectories GPU benchmarks use: - queue: "juliagpu" with cuda: "*" agent tags - No sandbox plugin (GPU passthrough limitation) - JULIA_CUDA_MEMORY_POOL='none' for accurate benchmarking 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
divital-coder
force-pushed
the
scimlbenchmarks-gpu-configuration
branch
from
daac9d9 to
5cf4b53
Compare
Author
|
@thazhemadam Unblock? |
Member
|
I unblocked |
Author
|
Thank you @ChrisRackauckas |
The UUID was incorrect, causing package resolution to fail. Correct UUID: 071ae1c0-96b5-11e9-1965-c90190d839ea 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Author
|
Do each of the commits require unblocking? I thought unblocking would unblock workflows for a specific PR! nvm, unblock again perhaps ? |
Member
|
yes each requires unblocking for this since it's changing the CI infrastructure |
- Replace @Assert with graceful CUDA_AVAILABLE check - Skip GPU benchmarks if CUDA.functional() returns false - Show CPU-only results when GPU unavailable - Prevent cascading errors from failed GPU initialization 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Author
|
unblock :) |
Member
|
I did |
Author
|
Does the JuliaGPU build queue on buildkite also sometimes run an agent without an available GPU? it keeps running on CPU still, due to cuda not being present. |
Member
|
yes, it needs to use a queue with gpus |
The ignore pattern `benchmarks/GPU/**` only matches files inside the GPU directory, not the directory path itself. When the path processor outputs `benchmarks/GPU` (the project directory), it wasn't being matched by the ignore pattern, causing GPU benchmarks to be picked up by the CPU launcher. Added `benchmarks/GPU` to the ignore list so both the directory name and its contents are properly ignored by the CPU launcher, ensuring GPU benchmarks only run via run_gpu_benchmark.yml on the juliagpu queue. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Author
|
unblocking required :) |
… queue The juliagpu queue agents (managed by JuliaGPU/buildkite) have these tags: - queue=juliagpu - cuda (with version or *) - cap (compute capability) - exclusive=true - gpu (GPU model) They do NOT have an `arch` tag. Requiring `arch: "x86_64"` caused jobs to wait indefinitely (8+ hours) because no agent matched all requirements. Verified against JuliaGPU/buildkite agent configurations (e.g., gpuci.17). Co-Authored-By: Claude Opus 4.5 <[email protected]>
Author
|
Hello, can someone unblock :) |
Member
|
Unblocked |
…g config
The `exclusive: true` tag was causing GPU benchmark jobs to wait indefinitely
(19+ hours) for an agent. Analysis of working SciML GPU CI configurations
(DiffEqGPU.jl) shows they do NOT use the exclusive tag.
The JuliaGPU buildkite agents already provide de-facto exclusivity via the
`--disconnect-after-job` flag - each agent runs one job then disconnects.
The explicit `exclusive: true` requirement is redundant and causes matching
issues.
New configuration matches DiffEqGPU.jl:
agents:
queue: "juliagpu"
cuda: "*"
Sources:
- DiffEqGPU.jl: github.com/SciML/DiffEqGPU.jl/blob/master/.buildkite/runtests.yml
- JuliaGPU agents: github.com/JuliaGPU/buildkite/blob/main/agents/
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
![NOTE] : Claude-code opus 4.5 with Github MCP was used to refine and open this PR.
Summary
This PR adds GPU benchmarking support to SciMLBenchmarks.jl:
run_gpu_benchmark.ymltemplate usingjuliagpuqueue with CUDA agentslaunch_benchmarks.ymlto route GPU benchmarks to GPU-specific pipelinebuild_benchmark.shwith GPU environment setup and CUDA verificationbenchmarks/GPU/directory with DiffEqGPU Lorenz ensemble benchmarkChanges
CI Configuration
.buildkite/run_gpu_benchmark.yml: New GPU benchmark template usingqueue: "juliagpu"withcuda: "*"agents (no sandbox plugin due to GPU passthrough complexity).buildkite/launch_benchmarks.yml: Added GPU watch block forbenchmarks/GPU/**paths, excluded GPU from CPU launcher.buildkite/build_benchmark.sh: Added GPU setup (JULIA_CUDA_MEMORY_POOL='none') and CUDA verification stepProof-of-Concept Benchmark
benchmarks/GPU/Project.toml: Dependencies (CUDA.jl, DiffEqGPU.jl, OrdinaryDiffEq.jl, StaticArrays.jl)benchmarks/GPU/EnsembleGPU_Lorenz.jmd: Lorenz system ensemble comparing CPU (EnsembleThreads) vs GPU (EnsembleGPUKernel) across 100-100000 trajectoriesTest plan
juliagpuqueue agentsjuliaecosystemqueue