Skip to content

Commit b48344a

Browse files
authored
Validate libtorch package architecture during binary validation (#8205)
## Problem The libtorch branch of `validate_binaries.sh` only downloaded and unzipped the package, then `exit 0` with no verification: ```bash if [[ ${MATRIX_PACKAGE_TYPE} == "libtorch" ]]; then curl "${MATRIX_INSTALLATION}" -o libtorch.zip unzip libtorch.zip exit 0 fi ``` So the arch mismatch in pytorch/pytorch#187812 -- where the Windows x86_64 libtorch package was overwritten by the arm64 build and shipped Aarch64 binaries under the x64 download URL -- slipped through validation entirely. ## Fix After unzip, read the machine field directly from the extracted `c10` library header (PE on Windows, ELF on Linux, Mach-O on macOS) and fail validation if it doesn't match the runner architecture. This is shared by the linux/windows/macos validate workflows, so all are covered. - Dependency-free: uses the runner's `python` and parses the binary header directly (no `file(1)`, which isn't reliably present on Windows runners). - Skips gracefully if the c10 library or python can't be found. The upstream packaging fix is pytorch/pytorch#187837; this is the validation-side guard so a future regression is caught before release rather than by users. ## Test Plan Verified header parsing against a real ELF binary and synthetic PE/ELF headers with arm64 machine fields: ``` real /bin/ls on x86_64 host -> x86_64 (matches, passes) synthetic AArch64 ELF (e_machine=0xB7) -> arm64 (mismatch on x64 runner, fails) synthetic ARM64 PE (machine=0xAA64) -> arm64 (mismatch on x64 runner, fails) ``` Confirmed the script still parses: `bash -n .github/scripts/validate_binaries.sh`. This change was authored with the assistance of an AI coding assistant (Claude Code). --------- Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>
1 parent 10e5555 commit b48344a

2 files changed

Lines changed: 80 additions & 2 deletions

File tree

.github/scripts/validate_binaries.sh

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -220,8 +220,14 @@ cleanup_conda_env() {
220220
handle_aarch64_cuda_override
221221

222222
if [[ ${MATRIX_PACKAGE_TYPE} == "libtorch" ]]; then
223-
curl "${MATRIX_INSTALLATION}" -o libtorch.zip
224-
unzip libtorch.zip
223+
LIBTORCH_PYTHON="python3"
224+
if [[ ${TARGET_OS} == 'windows' ]]; then
225+
# Windows runners only source conda.sh at this point without activating
226+
# an env, so no python is on PATH yet. Activate base to get one.
227+
conda activate base
228+
LIBTORCH_PYTHON="python"
229+
fi
230+
"${LIBTORCH_PYTHON}" "${SCRIPT_DIR}/validate_libtorch.py" "${MATRIX_INSTALLATION}"
225231
exit 0
226232
fi
227233

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/usr/bin/env python3
2+
"""Download, extract, and validate a libtorch binary package.
3+
4+
Validation loads the shipped c10 library with ctypes to confirm it is a
5+
working binary for the runner's platform and architecture. Loading a
6+
wrong-architecture library fails, so this catches a libtorch package being
7+
overwritten by a different-arch build on upload (see pytorch/pytorch#187812,
8+
where the Windows x86_64 package shipped Aarch64 binaries under the x64
9+
download URL).
10+
11+
Usage: validate_libtorch.py <download-url>
12+
"""
13+
14+
from __future__ import annotations
15+
16+
import argparse
17+
import ctypes
18+
import glob
19+
import os
20+
import shutil
21+
import sys
22+
import urllib.request
23+
import zipfile
24+
25+
26+
def find_c10_lib() -> str | None:
27+
for pattern in ("c10.dll", "libc10.so", "libc10.dylib"):
28+
matches = glob.glob(os.path.join("libtorch", "lib", pattern))
29+
if matches:
30+
return matches[0]
31+
return None
32+
33+
34+
def main() -> None:
35+
parser = argparse.ArgumentParser(description=__doc__)
36+
parser.add_argument(
37+
"url",
38+
nargs="?",
39+
default=os.environ.get("MATRIX_INSTALLATION", ""),
40+
help="libtorch package download URL",
41+
)
42+
args = parser.parse_args()
43+
if not args.url:
44+
sys.exit("ERROR: libtorch download URL not provided")
45+
46+
print(f"Downloading {args.url}")
47+
# Set an explicit User-Agent: the R2-backed CDN behind download.pytorch.org
48+
# returns 403 for the default "Python-urllib/x.y" agent.
49+
request = urllib.request.Request(
50+
args.url, headers={"User-Agent": "libtorch-validation"}
51+
)
52+
with urllib.request.urlopen(request) as response, open("libtorch.zip", "wb") as out:
53+
shutil.copyfileobj(response, out)
54+
with zipfile.ZipFile("libtorch.zip") as zf:
55+
zf.extractall(".")
56+
57+
lib = find_c10_lib()
58+
if lib is None:
59+
sys.exit("ERROR: c10 library not found under libtorch/lib")
60+
61+
# On Windows c10 resolves its sibling DLLs from the package lib directory.
62+
lib_dir = os.path.abspath(os.path.dirname(lib))
63+
if sys.platform == "win32" and hasattr(os, "add_dll_directory"):
64+
os.add_dll_directory(lib_dir)
65+
66+
print(f"Loading {lib} to validate it is a working binary for this runner")
67+
ctypes.CDLL(lib)
68+
print(f"Successfully loaded {lib}")
69+
70+
71+
if __name__ == "__main__":
72+
main()

0 commit comments

Comments
 (0)