Skip to content

New inference benchmarks #384

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Delaunay merged 91 commits into from
Feb 11, 2026
Merged

New inference benchmarks #384

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
62666f1
Improve issue reporting (#368)
Delaunay Sep 10, 2025
b58d6bb
Add job push button (#370)
Delaunay Sep 10, 2025
bd71e65
Tweak milabench realtime event tracking
Delaunay Sep 11, 2025
556aa28
Add unmerged code
Delaunay Sep 11, 2025
fe61b9f
Tweak to enable perfectly forwarding milabench events to external soures
Delaunay Sep 11, 2025
c3ebbe7
Make sure multinode jobs run on the right directory
Delaunay Sep 22, 2025
0096853
Merge branch 'master' of github.com:mila-iqia/milabench into staging
Delaunay Sep 22, 2025
8adb21c
Sync with upstream
Delaunay Sep 22, 2025
b24c96d
Merge branch 'staging' of github.com:mila-iqia/milabench into realtim…
Delaunay Sep 22, 2025
031f1b4
Use milabench utilities to create the milaench container_run cli
Delaunay Sep 23, 2025
3a250b1
Patch tqdm to avoid flodding logs with meaningless progress update
Delaunay Sep 24, 2025
7a7ca45
Try to ensure the logs are flushed when an issue happens to avoid slu…
Delaunay Sep 24, 2025
ece434a
Add timed log flush
Delaunay Sep 24, 2025
a1debb4
Tweak timed flush
Delaunay Sep 25, 2025
f1bd285
Add new inspection routes
Delaunay Sep 26, 2025
2e44c7a
Update script to use faster setup
Delaunay Sep 29, 2025
1495729
New shared_prepare
Delaunay Sep 29, 2025
4a57574
Tweak milabench global patch installation
Delaunay Oct 1, 2025
2ae3c96
Add SQL a valid metric pusher
Delaunay Oct 1, 2025
d408b34
Add SSH debug loging for tunnels
Delaunay Oct 7, 2025
df0cf52
Handle database reconnection gracefully
Delaunay Oct 7, 2025
b620c49
refactor configuration resolution (#376)
Delaunay Oct 15, 2025
bceabbd
Merge branch 'staging' of github.com:mila-iqia/milabench into realtim…
Delaunay Oct 15, 2025
ba5cbae
Client Server bench concept
Delaunay Oct 15, 2025
096141d
new vLLM inference benchmark
Oct 20, 2025
b0a5290
new vllm and whisper bench
Oct 21, 2025
2ae31ee
New inference benchmark for flux and whisper
Oct 24, 2025
febd754
Flux inference bench
Oct 28, 2025
945df31
New Text generation benchmark
Oct 28, 2025
b5e9f18
Normalize slurm configuration names
Delaunay Oct 30, 2025
ec208d5
Print a warning if we have no profile set
Delaunay Oct 31, 2025
e112891
fno_bench initial
Nov 4, 2025
651de2d
Move server code to the dashboard repository
Delaunay Oct 31, 2025
33d3401
additional fix:1
Nov 4, 2025
a0fbbd7
arg parsing
Nov 4, 2025
640d4d9
Fix some issues with milabench new not replacing some placeholder values
Delaunay Nov 4, 2025
f2b4828
Tweaks to huggingface environment folders
Delaunay Nov 4, 2025
1f6ba15
Toggle the inference benchmarks on
Delaunay Nov 4, 2025
2dbd528
Added some TimedIterator tests
Delaunay Nov 5, 2025
64b33ff
Add some checks to the timed iterator
Delaunay Nov 5, 2025
24bbe65
Merge branch 'fno_bench' of https://github.com/chelseajohn/milabench …
Delaunay Nov 5, 2025
b37265e
Merge branch 'fno_bench' of https://github.com/chelseajohn/milabench …
Delaunay Nov 5, 2025
4c654d1
Add the option to fetch the first batch in TimedIterator to reduce me…
Nov 6, 2025
3845f77
Adding milabench to ngc container
Delaunay Nov 6, 2025
0215093
Merge branch 'realtime_tracking' of github.com:mila-iqia/milabench in…
Delaunay Nov 6, 2025
9e2055b
Prepare tweaks
Delaunay Nov 10, 2025
1b7a230
Refactor to use generic huggingface download model and dataset
Delaunay Nov 13, 2025
14864f3
Tweak the prepare script to use no split by defaults
Delaunay Nov 14, 2025
41d9209
Add a new 'all' config
Delaunay Nov 17, 2025
5c9b20b
Force HF_HUB_CACHE to unify behaviour between benchmarks
Delaunay Nov 21, 2025
d8bc5d2
Pin dependencies for new benchmarks
Nov 24, 2025
6e1a76c
Avoid huggingface for Whisper inference
Nov 25, 2025
fc403d8
Tweak batch sizes
Delaunay Nov 27, 2025
ced512e
SPARK Tweaks
Nov 28, 2025
14ea143
vllm sweep concept
Dec 16, 2025
fd7bb62
Add a new IPMI monitor that starts and ends with milabench run
Dec 17, 2025
9fcdd23
Add new Kj energy spent estimate
Dec 17, 2025
75fb18a
Put real time attached to cuda event
Dec 17, 2025
c5f110f
Updated pin to torch==2.8
Dec 18, 2025
0ddde0c
Implement global throughput sampling
Dec 19, 2025
2ebfa61
update benchrun to match new pytorchrun API
Dec 19, 2025
974194c
Add new GPU Poll override
Dec 22, 2025
c521a90
New timeline script to display batch id
Dec 22, 2025
b3c7082
Add Energy stats guard on division by zero
Dec 22, 2025
16a4439
Tweak error report to not break exception trace lines
Dec 23, 2025
f2a5327
More robust stracktrace extraction
Dec 23, 2025
28400d0
Fix backward compatibility problem with pytorch 2.8 & 2.9
Dec 23, 2025
fec85e0
Update JAX libraries
Dec 23, 2025
283000f
Tweaks to support latest version of dependencies
Dec 23, 2025
1639c18
Make sure IP is set for the IPMI monitor
Dec 23, 2025
6ae5388
Tweaks for full run
Dec 23, 2025
d0bc811
-
Dec 23, 2025
c53edf1
Restore time estimate of the rate time
Dec 23, 2025
f160cfa
update gpu_poll to be float
Jan 5, 2026
b2f38f3
Gatehr tweaks
Jan 7, 2026
a9b4d73
New milabench event processor
Jan 7, 2026
10bad10
report tweaking
Delaunay Jan 7, 2026
4339308
Merge branch 'realtime_tracking' of github.com:milabench/milabench in…
Delaunay Jan 7, 2026
de4ea32
New reporting functions
Delaunay Jan 13, 2026
8373d64
Consolidate configs inside SystemConfig
Delaunay Jan 15, 2026
d813f6a
Tweak the unified config setting
Delaunay Jan 16, 2026
8ceabfd
Full milabench resume implementation
Delaunay Jan 19, 2026
fa4f3ed
add new dense llm sweep
Jan 30, 2026
0ad5235
Merge branch 'realtime_tracking' of github.com:milabench/milabench in…
Jan 30, 2026
4e913c3
Merge branch 'realtime_tracking' of github.com:milabench/milabench in…
Jan 30, 2026
1e2db1e
Ignore unmerged extension
Feb 3, 2026
b943774
Merge pull request #2 from milabench/realtime_tracking
Delaunay Feb 10, 2026
160cb39
Merge pull request #5 from milabench/refactor_config
Delaunay Feb 10, 2026
c828d27
Update code to use the new system structure
Feb 10, 2026
d8be139
Fix for docker not parsing the version file correctly
Feb 10, 2026
a1ebde2
Tweak IMPI monitor to be an op when not set
Feb 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,4 @@ benchmarks/*/src/

*.new.yml
*.png
fjobs_*.json
Loading
Loading