wheels: harden supply chain, slim wheels, rehearse PyPI publish#3072
Draft
hunhoffe wants to merge 32 commits into
Draft
wheels: harden supply chain, slim wheels, rehearse PyPI publish#3072hunhoffe wants to merge 32 commits into
hunhoffe wants to merge 32 commits into
Conversation
Generate two pip-compile lockfiles with --generate-hashes so every
package the wheel build pulls from PyPI is verified against a
sha256 hash before install:
- requirements.lock — full transitive resolution of the existing
requirements.txt (setuptools, nanobind, cmake, ninja, numpy,
pybind11, wheel, importlib-metadata, pip, dataclasses, rich).
- ci-tools.in / ci-tools.lock — auxiliary tools currently
pip install-ed ad-hoc in the workflows (auditwheel, delvewheel,
patchelf, pkginfo).
Wire cibuildwheel to consume the lockfile by:
- swapping the before-build `pip install -r requirements.txt` for
`pip install --require-hashes -r requirements.lock` (Linux/macOS
/Windows blocks),
- adding `build-frontend = { name = "pip", args = ["--no-build-isolation"] }`
so the actual wheel build re-uses the hash-verified deps rather
than re-fetching [build-system].requires under PEP 517 isolation.
Without --no-build-isolation the require-hashes install is decorative:
pip would still pull setuptools/nanobind fresh in the isolated build
env. With it, the wheel build is reproducible bit-for-bit (modulo
timestamps) given the same lockfile.
Refresh recipe (also picked up by the Dependabot config in the
parallel add-dependabot-scorecard PR):
pip install pip-tools
pip-compile --generate-hashes --allow-unsafe --strip-extras \\
-o requirements.lock requirements.txt
pip-compile --generate-hashes --allow-unsafe --strip-extras \\
-o ci-tools.lock ci-tools.in
Local verification: tampering test (zero out all hashes; pip
correctly rejects with "Expected ... Got ..." mismatch).
Leaves [build-system].requires loose-lower-bounds for sdist metadata
extraction (consulted by tooling that does `pip install <sdist>`,
not by our actual wheel build).
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
Swap every ad-hoc `pip install <tool>` in the wheel-build workflows for `pip install --require-hashes -r requirements.lock` (build-time deps) or `... -r ci-tools.lock` (auxiliary tools), so a compromised release of any of these packages is rejected by hash mismatch rather than silently incorporated into the produced wheel. buildRyzenWheels.yml: - Linux build: prepend hash-required installs before pip wheel. - Windows build: same. mlirAIEDistro.yml: - Windows wheel-repair (delvewheel) and get_wheel_version (pkginfo) steps now read from ci-tools.lock. - Aarch64 build path: requirements.txt -> requirements.lock with --require-hashes; --no-build-isolation added to pip wheel for consistency with the cibuildwheel paths now configured the same way in pyproject.toml. The wheels themselves are unchanged in shape; this only tightens where the ingredients come from. Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
Two CI failures from the prior commit:
1. Linux 3.10 builds failed:
ERROR: Could not find a version that satisfies numpy==2.4.4
(versions: ..., 2.2.6)
Cause: I generated requirements.lock against Python 3.12, so
pip-compile picked numpy==2.4.4 (3.11+-only). The wheel matrix
covers cp310-cp314, so the lockfile must resolve for the lowest.
Fix: regenerate with --python-version 3.10. numpy now pins to
2.2.6, which installs cleanly on every Python 3.10-3.14.
2. All Windows builds failed:
ERROR: Failed building wheel for patchelf
Cause: ci-tools.lock pinned patchelf unconditionally; patchelf
has no Windows wheel, so pip tried to build it from sdist.
Fix: add platform markers to ci-tools.in (auditwheel/patchelf
Linux-only, delvewheel Windows-only, pkginfo cross-platform).
Tooling change: switch the lockfile generator from pip-tools to uv.
pip-compile resolves only for the host platform, which silently
drops the delvewheel pin when run on Linux. `uv pip compile
--universal` resolves cross-platform and emits a single lockfile
with markers covering both. Updated the in-file refresh recipe.
The CI workflows themselves are unchanged from the prior commit.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
The previous commit's switch to `uv pip compile --universal` solved the cross-platform-resolution problem (delvewheel-on-Linux drops it from the lock), but added uv as a new tool the maintenance workflow depends on. Backing that out: - ci-tools.in: drop the `; platform_system == "Windows"` marker on delvewheel. delvewheel ships as a universal `py3-none-any` wheel, so installing it on Linux is harmless (~50 KB unused). With no Windows-only entry left, plain pip-compile from a Linux host resolves the file correctly. auditwheel + patchelf keep their Linux markers (those genuinely have no Windows wheel). - requirements.lock: regenerated with pip-compile inside a Python 3.10 container (`docker run --rm python:3.10-slim ...`). Numpy still pins to 2.2.6 — the matrix-Python fix from the prior commit is unchanged. The refresh recipe in ci-tools.in is updated back to plain pip-compile, with a one-line note explaining why delvewheel is intentionally unmarked. Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
Resolves the OpenSSF Scorecard Pinned-Dependencies finding for the githubAction bucket: all `uses:` references in .github/workflows/ now point at a full 40-char commit SHA with the human-readable tag retained as a trailing comment. Dependabot's existing actions-minor-patch group reads the comment to surface bumps as readable PRs while rewriting the SHA. Pre-existing bare-SHA pins for aminya/setup-cpp (v1.8.0) and llvm/actions/install-ninja (untagged, tracks main) get explanatory comments so dependabot has something to compare against.
Resolves the Scorecard Token-Permissions finding: each workflow gets a top-level default-deny `permissions: contents: read`. Jobs that need write access (release uploads, suggester comments, gh-pages deploys) either already had explicit job-level permissions blocks or get one added (mlirDistro.upload_distro_wheels gains `contents: write` for the ncipollo release-action call). scorecard.yml, pruneReleaseAssets.yml, and update-llvm.yml already had explicit permissions and are left alone.
Adds a pip-compile-generated requirements_dev.lock with --generate-hashes covering all 44 transitive deps. All 15 CI sites that previously ran `pip install -r python/requirements_dev.txt` now use `pip install --require-hashes -r python/requirements_dev.lock` — tampering produces a hash mismatch instead of a silent swap. requirements_dev.txt stays loose for local developers (iron_setup.py still reads it). Dependabot's existing /python ecosystem entry covers the new lockfile.
Both setup.py files now copy the upstream LICENSE next to the build script at setup time (mlir-aie repo's LICENSE for mlir_aie_wheels; llvm-project/LICENSE.TXT for mlir_wheels) and declare it via `license_files`, so setuptools places it under `<dist>.dist-info/` where pip and PyPI tooling can find it. Auto-copied LICENSE is gitignored to keep the source tree clean.
Both wheels now declare `project_urls` so PyPI renders Source / Issues / Documentation links in the sidebar. utils/mlir_wheels/setup.py also gains the missing `url`, `license`, and `classifiers` fields it was silently omitting, and switches author to AMD to match the mlir_aie_wheels package metadata.
Mirrors the hash-pinning treatment from utils/mlir_aie_wheels onto the LLVM distro wheel: * requirements.in + requirements.lock (pip-compile --generate-hashes) cover the [build-system].requires set except mlir-native-tools, which is intentionally left loose because it lives on an internal index reached via PIP_FIND_LINKS — pip-compile can't resolve it against PyPI. * ci-tools.in + ci-tools.lock pin the wheel-repair / metadata-reader utilities (delvewheel, pkginfo) previously installed ad-hoc. * pyproject.toml adopts the same --no-build-isolation pattern so PEP 517 reuses the pre-installed hash-verified deps instead of re-fetching [build-system].requires fresh from PyPI. * mlirDistro.yml swaps the two remaining ad-hoc `pip install` lines for --require-hashes reads of the new lockfiles.
After each wheel is built and repaired, run `twine check --strict`
against the final artifact. Catches malformed README markdown (which
otherwise renders broken on PyPI), missing or wrong classifiers, and
invalid version strings — the exact failure modes that surface only
when a wheel hits a real package index.
Adds `twine` to both wheels' ci-tools.{in,lock} so the validator is
installed alongside delvewheel/auditwheel/pkginfo via the existing
--require-hashes path.
Strict mode treats warnings as failures: lets us catch metadata drift
before any PyPI upload pipeline is wired in.
buildRyzenWheels previously had no post-build sanity check that the wheel can be installed and imported. Add a step in both build-repo (Linux) and build-windows that creates a fresh venv with the matrix Python, `pip install`s the freshly-built wheel, and runs `import aie.ir / aie.extras / aie.helpers` — the same import probes mlirAIEDistro's smoke_test_wheels uses, but exercising the real pip-install path so packaging bugs in entry_points, install_requires, or package_data fail loudly before the wheel reaches the Releases page.
Each wheel-build job now runs actions/attest-build-provenance after the twine check, producing a SLSA-Build-L3 attestation tied to the workflow run that built it. The attestation lives in GitHub's transparency log and can be verified offline with `gh attestation verify <wheel> --repo Xilinx/mlir-aie` even without a PyPI publishing path. When the deferred publish workflow lands, pypa/gh-action-pypi-publish picks up the same attestation and forwards it to PyPI so the project page shows the verified-build badge with zero extra config. Skipped on pull_request because fork PRs cannot mint id-token writes; also skipped on mlirDistro's schedule trigger (matches the existing upload-wheels gating). Each affected job gains id-token: write and attestations: write at the minimum scope required by the action.
Both tools are editor/inspection utilities: aie-lsp-server is an MLIR language server consumed by VSCode/Emacs at edit time, aie-visualize draws AIE device layouts from IR. Neither is on the runtime path that aiecc uses to actually compile, and the in-tree usage is one lit test for the LSP and zero references for visualize. Together they contributed ~186 MB uncompressed (~55 MB compressed) to the wheel — meaningful slice of the path back under PyPI's 100 MB cap. Gated by two new CMake options (AIE_BUILD_LSP_SERVER, AIE_BUILD_VISUALIZE) defaulting to ON so dev builds keep both binaries and the LSP hover lit test keeps passing. utils/mlir_aie_wheels/setup.py passes OFF when building the wheel; the existing add_lit_testsuite TEST_DEPENDS and the LSP test gain matching gates so a deliberately minimal local build still has a clean lit run. Removes the now-broken console_scripts entries for both tools from the wheel metadata so users don't end up with shims pointing at binaries that aren't shipped.
Every build now rehearses what a PyPI upload would look like: takes a copy of each repaired wheel, strips ELF .so (--strip-unneeded preserves dynamic symbol table for dlopen) and bin/ executables (--strip-all is fine, nothing dlopens these), repacks, runs twine check --strict against the stripped variant, and reports unstripped + stripped sizes and the 100 MB PyPI default cap verdict to GITHUB_STEP_SUMMARY. Stripped wheels upload as the `*_stripped_for_pypi` CI artifact so they can be inspected offline. The unstripped wheel continues to GitHub releases as before — unchanged user-facing behavior; the rehearsal is purely additional. Once a real publish path lands, swap the artifact upload for pypa/gh-action-pypi-publish in the same job — the strip pipeline is already producing the right input. The strip ratio is interesting on its own: on the 2026-05-20 release wheel, stripping pulls 197 MB out of the unpacked tree but only 44 MB off the compressed .whl (wheel-zip's deflate was already capturing most of the symbol-table redundancy). Useful baseline for the size-increase request to PyPI admins.
The PyPI rehearsal's GITHUB_STEP_SUMMARY table already records every unstripped/stripped/reduction number per run, durably on the workflow run page. The stripped wheels themselves were uploaded as ~200 MB CI artifacts per matrix combo — burns Actions storage quota for data anyone can reproduce in seconds by re-running the rehearsal script against the unstripped wheel. Keep the rehearsal itself (twine check, size verdict, summary table); drop just the upload-artifact steps and update the summary footer to match.
23 archive files in mlir_aie/lib/*.a totaling 49 MB uncompressed (~10 MB compressed) are only useful for downstream C++ code that links against AIE / MLIR symbols at native build time. Anyone who pip installs mlir-aie reaches the toolchain through the Python API and never touches these. Delete them between cmake --install and the wheel-pack step in the mlir_aie wheel build. The mlir LLVM-distro wheel (utils/mlir_wheels/) is unaffected — its audience overlaps more with native-toolchain consumers, so keep its dev libs.
Plain binutils \`strip\` only knows ELF — it silently no-ops on PE files, so Windows wheels were getting reported as 0%-reduction without any sign that the strip step had been ineffective on them. Switch to llvm-strip (binary-format-agnostic; handles ELF .so and PE .pyd/.dll/.exe in the same invocation). Auto-detect: prefer unversioned llvm-strip, fall back to highest-numbered llvm-strip-NN, fall back to binutils strip with a warning. Measured side effect: the post-switch rehearsal still reports 0% reduction on the current MSVC-built Windows wheels, but now we know that's actually true rather than a tooling miss — the DLLs have no \`.debug_*\` sections to remove (MSVC puts debug info in separate \`.pdb\` files that aren't shipped). The Windows wheel is at its natural floor; further reductions there need an upstream linker/build change, not post-link stripping.
CMake was shipping three things into the wheel that nobody should ever receive: * mlir_aie/src/ (~1.7 MB) — declare_mlir_python_sources staging directories, byte-for-byte duplicates of mlir_aie/python/ * mlir_aie/lib/objects-Release/ (~2.1 MB) — 19 intermediate .o files from the LLVM build * **/__pycache__/ (28 .pyc files, ~0.6 MB) — Python bytecode caches generated by any python invocation that happened during build Clean them up post-install. ~4.4 MB uncompressed off the wheel (~1.5 MB compressed). More importantly: end-users should not be receiving build artifacts at all, regardless of size.
Same audience filter we applied to lib/*.a: anyone who pip installs
mlir-aie reaches the toolchain through the Python API and never touches
the C++ MLIR-pass-author headers, the C API headers, or the
find_package(AIE) cmake configs.
Dropped:
include/aie/ - 11 MB, MLIR C++ pass-author headers
include/aie-c/ - 24 KB, C API
include/bootgen_c_api.h - 4 KB
include/xaienginecdo_static/ - 6.5 MB, paired with the static archives
already dropped (headers without their
matching .a libs are unusable anyway)
lib/cmake/ - 92 KB, find_package configs
Kept:
include/aie_api/ - 4.3 MB, user kernels #include "aie_api/aie.hpp"
include/aie_kernels/ - 740 KB, user kernels #include kernel utils
lib/regdb/ - 2.7 MB, aie-opt reads at runtime
~18 MB uncompressed (~5 MB compressed) off the wheel. End-user behavior
of `aiecc my_design.mlir` is unchanged.
The LLVM distro wheel pipeline originated with Maksim Levental; keeping author= preserves the historical attribution. AMD is the current upstream owner of the mlir-aie repo that drives ongoing maintenance, so add it as maintainer= rather than overwriting author.
The post-install cleanup blocks were carrying redundant inline annotations that just listed what the dev_paths literal already names. Collapse the static-archives loop into the same dev_paths list (one audience filter, one pass) and shrink the comment to the actual "why". Also relocate the LICENSE materialization from the module top to immediately before setup(), so the side-effect-at-import sits next to the license_files= consumer that needs it. Cannot move into CMakeBuild.run() — egg_info reads license_files before build_ext fires.
hunhoffe
force-pushed
the
dep-pinning-branch
branch
from
ac33d2f to
cd50c7f
Compare
hunhoffe
changed the title
The hash-locked requirements_dev install was failing on Windows runners with `ERROR: Failed building wheel for patchelf`. patchelf is pulled in transitively by cibuildwheel via auditwheel; it has no Windows binary wheel and falls over building from source on cl.exe because its CMake invokes `.\configure`. Add an explicit `patchelf; platform_system == "Linux"` line to requirements_dev.txt and regenerate the lockfile so pip-compile propagates the marker. The loose `pip install -r ...txt` install never hit this because pip resolved a different cibuildwheel transitive set; under --require-hashes we have to honor whatever's in the lock. Fixes the Compile-across-platforms windows-2022 failures introduced by the b298ff8 hash-pin commit.
The default-deny `permissions: contents: read` we added at the top of each workflow zeros out every unlisted permission — including `packages`, which run-on-arch-action needs to pull ghcr.io/xilinx/mlir-aie/* base images. The Ryzen-AI-Software jobs caught this first with a 403 Forbidden on the manifest HEAD request. Add `packages: read` at the workflow level for the four files that pull from ghcr (buildAndTestAieTools, buildAndTestAieToolsHsaBuildOnly, buildAndTestRyzenAISw, buildRyzenWheels). Still least-privilege — read-only, doesn't widen any job's surface beyond what the registry pull needs. Fixes the Ryzen-AI-Software failures introduced by b6c0865.
Vanilla pip-compile only sees the host platform / Python — Linux + py3.12
in our case — so platform-conditional or python-version-conditional
transitive deps were silently absent from the locks. CI install on
Windows / older Pythons then tripped --require-hashes when pip resolved
deps the lock didn't cover:
- backports.tarfile (jaraco-context's py<3.12 conditional via twine):
failed on Linux py3.11 wheel build
- colorama (build package's Windows conditional): failed on every
Windows wheel-build matrix entry
- patchelf (cibuildwheel's transitive, Linux-only): already gated
manually in the previous fix; uv carries it through with the same
semantics plus the darwin variants cibw also wants
Regenerate via `uv pip compile --universal --python-version=3.11` so the
locks include the union of every (python-version × OS) combination
consumers might hit. Markers in the lock select the right entries at
install time on each platform.
Separately, revert the Ryzen-AI-Software workflow's pip-install of
requirements_dev to the loose .txt — that runner is intentionally
cp310-only for the Ryzen AI 1.3 SDK, and the lock pins cibuildwheel 3.x
(py>=3.11 only). The lock works everywhere else; this one runner stays
loose by SDK constraint.
CI on the hash-locked dev env upgraded numpy to 2.4.6 (latest matching), which exposed two NumPy-1.x-only patterns in the runtime/test code: * python/utils/hostruntime/xrtruntime/tensor.py:61 used np.array(..., copy=False). NumPy 1.x: avoid copy if possible. NumPy 2.x: must not copy, raise ValueError if would have to. Switch to np.asarray(..., dtype=...) which has identical "avoid copy when possible, copy when necessary" semantics on both major versions. * programming_examples/basic/event_trace/test.py at lines 261 and 285 called np.frombuffer on the numpy array returned by pyxrt's bo.read(). NumPy 2.x rejects non-C-contiguous buffers with "ndarray is not C-contiguous". Wrap the source in np.ascontiguousarray() — zero copy when already contiguous, one defensive copy otherwise. Both fixes are backward-compatible with NumPy 1.x, so users still on the runtime numpy<2.0 constraint see no change.
Two distinct Windows wheel-build failures, fixed together because they both live in the cibw before-build pipeline: * numpy was unpinned in both wheels' input requirements; uv resolved to 2.2.6 which has no cp314 Windows binary wheel, and pip then tried building numpy from source on the Windows runner, where meson found Git for Windows' link.exe before MSVC's link.exe and failed. Pin numpy>=2.3 in both inputs so uv picks a version (2.4.6) that ships cp314-win wheels. * `pip install --require-hashes -r requirements.lock` on Windows can fail with "To modify pip, please run the following command: ...python.exe -m pip install ..." because the lock pins `pip` itself, and pip refuses to self-update via `pip install` on Windows. Switch every cibw before-build line to `python -m pip install` in both pyproject.toml files (Linux/macOS/Windows for symmetry — the canonical invocation works the same everywhere). Fixes Windows 3.14 wheel build for both mlir_aie and mlir distro, and the mlirAIEDistro Windows pip-self-modify error.
The previous fix switched cibw before-build to python -m pip install, but missed the smoke-test step's `<venv>/Scripts/pip install --upgrade pip` line, which trips the same Windows "ERROR: To modify pip, please run the following command: ...python.exe -m pip install ..." check. Switch both smoke test steps (Linux + Windows) to the python -m pip form for symmetry.
The two earlier commits switched cibw before-build and the smoke test to `python -m pip install ...`, but missed the 24 remaining bare `pip install --require-hashes -r ...lock` calls scattered across the workflow YAMLs. Most live on Linux runners where bare `pip install` works fine, but several land in the Windows wheel job paths where pip refuses to self-update via bare `pip install` (the lock pins pip itself). Last failure was the mlirAIEDistro Repair-wheels-Windows step. Switch all 24 to `python -m pip install` for symmetry. Linux behavior unchanged; Windows now consistently uses the form that bypasses the self-modify check.
My previous sweep regex matched 'pip install --require-hashes' even when already preceded by 'python -m ', producing 6 lines of the form 'python -m python -m pip install --require-hashes ...'. mlirDistro caught this immediately: the workflow itself failed to start because the run line was malformed. Collapse the doubled prefix back to single 'python -m'.
`upload_distro_wheels` has had `permissions: { id-token: write, contents:
write }` since 8de2b28. My b6c0865 permissions sweep missed the
existing block and added a redundant `permissions: { contents: write }`,
which GHA rejected with "permissions is already defined" — silently
blocking the whole workflow file from starting any jobs.
Drop the redundant block; the pre-existing one already grants the needed
contents:write (plus id-token:write for any future attestation).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Originally this PR just hash-pinned the mlir_aie wheel's build deps. It grew to cover three related threads that all needed to land before moving towards PyPI publication.
Supply chain. Hash-pin every pip dep we touch (mlir_aie wheel, mlir distro wheel,
python/requirements_dev.txt). SHA-pin every GitHub Action with a# vX.Y.Zcomment so dependabot keeps maintaining them. Default-denypermissions: contents: readon workflows; widen at job level only where needed. Addactions/attest-build-provenanceon every wheel sogh attestation verifyworks end-to-end, even before any PyPI upload exists.Wheel hygiene. Ship
LICENSE(it was missing), setproject_urls/classifiersso the PyPI page renders properly, fix author/maintainer attribution on the mlir distro wheel. Drop ~70 MB of dev-only content from the wheel:aie-lsp-serverandaie-visualize(editor tools), static.aarchives, C++ headers, cmake configs, plus a handful of CMake-leaked artifacts that shouldn't have been shipping at all (src/duplicates,lib/objects-Release/.o files, stray__pycache__).PyPI rehearsal. Each build now strips a copy of the wheel, runs
twine check --strict, and writes a size + cap-verdict table to the run summary. No upload happens. The wheel still goes to GitHub releases unstripped as before — the rehearsal just gives us measured data for the eventual size of wheels we'd like to publish to PyPI.A few things worth flagging:
aie-lsp-server/aie-visualizestill build by default for local dev (newAIE_BUILD_LSP_SERVER/AIE_BUILD_VISUALIZECMake options, bothON); only the wheel build passesOFF.