Skip to content

Report torch/torchvision versions and binary sizes for every validated build - #8613

Open
atalman wants to merge 3 commits into
mainfrom
atalman/validate-binaries-size-report
Open

Report torch/torchvision versions and binary sizes for every validated build#8613
atalman wants to merge 3 commits into
mainfrom
atalman/validate-binaries-size-report

Conversation

@atalman

@atalman atalman commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Every validated build now reports what it installed and how big it is.

Why

check_wheel_size exists to enforce an 850 MB ceiling, and its scope is deliberately narrow: linux and linux-aarch64 pip wheels only, skipping ROCm and libtorch. That means windows, macos and ROCm builds report no size at all, and even where the check does run the number only appears as an annotation.

What this adds

A write_build_report step after the smoke tests, on every OS, that never fails the job:

--- Build report
| field | value |
| --- | --- |
| build | linux / py3.12 / cuda 12.8 |
| torch | 2.10.0.dev20260824+cu128 |
| torch wheel | 812.4 MB |
| torch installed | 2143.7 MB |
| CUDA | 12.8 |
| cuDNN | 9.24.0 |
| NCCL | 2.30.7 |
| torchvision | 0.25.0.dev20260824+cu128 |
| torchvision wheel | 8.1 MB |
| torchvision installed | 24.3 MB |

Two sizes, because they answer different questions:

  • wheel -- compressed download size, what users actually pull from the index. Read off pip's Downloading/Using cached line, so it is - on the uv/wheel-variants path and when the wheel was already satisfied.
  • installed -- unpacked bytes on disk, walked from the installed package, so it is available on every install path and every OS.

Versions are read from the installed packages rather than from the matrix, so a mismatch between what was requested and what pip resolved shows up in the report.

The report goes to stdout always, and is appended to the job summary when the runner exposes a writable one -- it is not reachable from inside the validation container, so that append is guarded rather than assumed.

Refactor

The size parsing is factored out of check_wheel_size into parse_wheel_size_mb <log> <dist> and reused for both torch and torchvision. check_wheel_size keeps its current scope, threshold and behaviour; only the parsing moved.

Not covered

  • libtorch builds return before this point (they go through validate_libtorch.py), so they still report nothing. Worth a follow-up if libtorch size matters.
  • The report is per job. If a single aggregated table across all ~170 builds is wanted, that needs an artifact per job plus a final aggregation job -- deliberately left out of this change.

Test plan

  • bash -n clean.
  • parse_wheel_size_mb unit-checked against a captured pip log: torch -> 812.4, torchvision -> 8.1, absent distribution -> empty (torch-[0-9] does not match torchvision-/torchaudio-).
  • cuDNN integer decoding checked: 91002 -> 9.10.2, 92400 -> 9.24.0, 91701 -> 9.17.1, matching the values published in pytorch RELEASE.md.
  • write_build_report with no torch installed renders import failed: ... and exits 0 rather than aborting the job.
  • With GITHUB_STEP_SUMMARY pointing at an unwritable path, the function emits zero bytes on stderr and still exits 0.

…d build

check_wheel_size only measures linux and linux-aarch64 pip wheels, skips ROCm
and libtorch, and exists to enforce a ceiling -- so windows, macos and ROCm
builds report no size at all today.

Add a per-build report that runs on every OS and never fails the job. It prints
two sizes, because they answer different questions: the compressed wheel
download size (what users pull from the index, read off pip's output) and the
unpacked installed size (measured from the installed package, so it is
available on every install path including uv/wheel-variants). Versions come
from the installed packages rather than the matrix, so a mismatch between what
was requested and what pip resolved is visible.

The wheel-size parsing is factored out of check_wheel_size into
parse_wheel_size_mb and reused; the ceiling check keeps its current scope and
behaviour.
@vercel

vercel Bot commented Aug 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
torchci Ignored Ignored Aug 25, 2026 1:33am

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 24, 2026
Comment thread .github/scripts/validate_binaries.sh Outdated
report=$(cd "${TMPDIR:-/tmp}" 2>/dev/null || cd "${HOME}"; "${PYTHON_RUN}" - \
"${TARGET_OS}" "${MATRIX_PYTHON_VERSION:-?}" "${MATRIX_GPU_ARCH_TYPE:-cpu}" \
"${MATRIX_GPU_ARCH_VERSION:-}" "${torch_wheel_mb}" "${vision_wheel_mb}" <<'PY'
import os

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be better to have this as a standalone script, so lint and other rules would apply and it would be easier to maintain the quality.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, done in 3edf8f3 — it is now .github/scripts/build_report.py.

That puts it under FLAKE8 and PYFMT (**/*.py) and MYPY (.github/scripts/**/*.py). mypy.ini gets one exemption, following the existing per-module sections in that file:

[mypy-.github.scripts.build_report]
# torch/torchvision are not installed in the lint environment; the script
# imports them defensively at runtime and degrades when they are absent.
disable_error_code = import-not-found, import-untyped

One bonus that only became clear while moving it: the heredoc version had to cd out of the repo before running python, because python - puts the cwd on sys.path and the validation runs from the pytorch/pytorch checkout — so import torch could pick up the source tree instead of the installed package. A script file makes sys.path[0] the script's own directory, so that hazard is gone and the cd with it.

validate_binaries.sh is 73 lines lighter; the shell function now just invokes the script and handles the job-summary append.

Per review: embedded in a bash heredoc the python was invisible to flake8,
mypy and pyfmt. As .github/scripts/build_report.py it is covered by all three
(mypy.ini gets an import-not-found exemption, since torch is not installed in
the lint environment).

Being a real script file also removes the reason the caller had to cd out of
the repo first: sys.path[0] is now the script's own directory rather than the
cwd, so `import torch` can no longer pick up the pytorch source checkout the
validation runs from.
The [mypy-.github.scripts.build_report] section never matched: with no
__init__.py in that directory mypy names the module build_report, not the
dotted path, so import-not-found was still reported for torch and
torchvision.

Use ignore_missing_imports on the imported modules instead, which is how mypy
resolves third-party modules that are absent from the lint environment.
Verified with mypy 1.13.0 (the pinned version) against mypy.ini: clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants