Build a repo that repackages official LLVM releases into per-platform, range-fetchable artifacts

[marsninja]

Build a repo that repackages official LLVM releases into per-platform, range-fetchable artifacts

You are working in a fresh git repository. gh is authenticated. Build the project described below. Read the whole brief before writing anything, then follow the implementation order at the end.

---

1. Context and goal

The official LLVM releases (github.com/llvm/llvm-project/releases) ship each platform as a single
clang+llvm-<version>-<triple>.tar.xz. These are solid xz streams: there is no seekable index, so to
get any file out you must download and decompress the whole multi-GB stream. Downstream projects that link
only a subset of LLVM's static libraries still have to pull the entire tarball.

This repo fixes that without rebuilding LLVM. For each LLVM release and each platform LLVM already
publishes, it:

1. Downloads the official clang+llvm-* tarball.
2. Extracts only the development surface (static libs, headers, CMake package files, llvm-config).
3. Repackages that surface as a plain ZIP (DEFLATE, with a central directory) so any HTTP client can
   pull individual members via Range requests — this is the whole point; tar.xz cannot do this, zip can.
4. Generates a dependency manifest (JSON) so a consumer can compute the transitive closure of the libs
   it needs and fetch only those members.
5. Publishes the zips + manifests as GitHub Releases on this repo, mirroring upstream's version/platform
   matrix.

The value this repo adds over the several existing "prebuilt LLVM static libs" repos is specifically the
granular, range-fetchable access plus dependency metadata — not yet another full build.

2. Non-goals (do not do these)

• Do not build or compile LLVM. No cmake/ninja of llvm-project. We only consume official tarballs.
• Do not carry patches or change LLVM build options.
• Do not invent a new archive format. Plain ZIP is the container. (SOZip is unnecessary here because the
  members are many separate .a/.lib files; per-member random access from the zip central directory is
  enough. Note this in the README but don't implement it.)
• Do not hardcode the platform list (see §3).

3. Platform support strategy ("all platforms LLVM ships")

Discover platforms dynamically, never hardcode them:

• For a given LLVM version, query the upstream release assets with gh api (e.g.
  repos/llvm/llvm-project/releases/tags/llvmorg-<version>) and select assets whose names match
  clang+llvm-<version>-<triple>.tar.xz. The set of <triple>s varies by release; whatever upstream
  published is what we process.
• A "latest" mode should resolve the newest non-prerelease llvmorg-* tag.

Key property to exploit: repackaging is execution-free. We never run the platform's binaries; we only
read text/CMake files and repackage files. So every platform (Linux/macOS/Windows, x86_64/aarch64/etc.)
can be processed on a single ubuntu-latest runner. Build a dynamic matrix (one job per discovered
triple) so jobs parallelize and retry independently, but they all run on Linux.

Handle these per-platform shapes:

• Windows tarballs use .lib not .a, llvm-config.exe, and backslash-free zip paths (normalize to /).
• Some tarballs may ship only a shared libLLVM and lack the full set of component .a/.lib files, or
  lack lib/cmake/llvm/. If the dev surface is missing, skip that platform with a recorded warning in
  the index (don't fail the whole run).

4. The cross-platform manifest trick (most important correctness point)

Do not run llvm-config to compute dependencies — you can't execute a foreign-arch/OS binary on the
Linux runner, and that would break the "all platforms from one runner" property. Instead, parse the CMake
package files, which are plain text and identical in format across platforms, found under
lib/cmake/llvm/:

• LLVMConfig.cmake — read LLVM_AVAILABLE_LIBS for the full set of LLVM library target names, and capture
  LLVM_TARGETS_TO_BUILD, include dirs, and version.
• LLVMExports.cmake — for each add_library(<Target> STATIC IMPORTED), read its
  INTERFACE_LINK_LIBRARIES. These edges give the inter-library dependency graph. Entries that are other
  LLVM targets are internal deps (resolve to files in the zip); entries that are system libs / link
  flags (e.g. -lpthread, z, zstd, m, dl, ZLIB::ZLIB) are external requirements the consumer
  must satisfy on their own link line.
• LLVMExports-release.cmake (config-specific; the config suffix may vary) — read IMPORTED_LOCATION_* to
  map each target name to its actual on-disk file (e.g. LLVMCore → lib/libLLVMCore.a).

The v1 manifest is keyed by library target name. Friendly llvm-config component aliases (e.g.
engine, native, all-targets, core) are not in the CMake files; treat them as an optional
stretch goal — if you implement them, do so only by running the host-matching tarball's native
llvm-config --components/--libs (e.g. on the x86_64 Linux tarball) and shipping that alias map as a
best-effort extra, clearly labeled as host-derived.

Manifest schema (manifest.json, one per platform)

{
  "schema_version": 1,
  "llvm_version": "20.1.8",
  "triple": "x86_64-linux-gnu-ubuntu-22.04",
  "zip_asset": "llvm-20.1.8-x86_64-linux-gnu-ubuntu-22.04-dev.zip",
  "zip_sha256": "…",
  "upstream_tarball": "clang+llvm-20.1.8-x86_64-…tar.xz",
  "upstream_sha256": "…",
  "include_prefix": "include/",          // where headers live inside the zip
  "cmake_prefix": "lib/cmake/",
  "libs": {
    "LLVMCore": {
      "file": "lib/libLLVMCore.a",       // path inside the zip, or null if header-only/missing
      "size": 12345678,
      "sha256": "…",
      "deps": ["LLVMBinaryFormat", "LLVMRemarks", "LLVMSupport", "LLVMTargetParser"],
      "external": ["-lpthread", "-lz"]    // non-LLVM link requirements passed through
    }
    // … every entry in LLVM_AVAILABLE_LIBS
  }
}

Also emit a top-level index.json per release listing every platform, its zip asset name + URL +
sha256, its manifest asset name, and any skipped platforms with reasons.

5. Artifact layout per platform

Inside each llvm-<version>-<triple>-dev.zip, include:

• lib/*.a (or *.lib on Windows) — the static component libraries.
• include/ — all headers (needed wholesale; do not try to slice headers).
• lib/cmake/ — the CMake package files (so find_package(LLVM) can work after extraction).
• bin/llvm-config (or .exe) if present.
• manifest.json — also embed a copy inside the zip for convenience.

Publish per release as GitHub Release assets:

• llvm-<version>-<triple>-dev.zip (one per platform)
• llvm-<version>-<triple>-manifest.json (standalone, so a consumer fetches a few-KB file to plan the
  closure before touching the zip)
• index.json

Use the release tag v<llvm-version> (e.g. v20.1.8) on this repo.

6. Consumer CLI (llvm-slice)

Ship a small, dependency-light Python CLI in client/ that proves the whole thing works end to end:

• llvm-slice list --version 20.1.8 --triple <t> — fetch the standalone manifest, print libs.
• llvm-slice resolve --version 20.1.8 --triple <t> --libs LLVMOrcJIT,LLVMX86CodeGen — compute and print
  the transitive closure (internal deps) plus the merged external link requirements.
• llvm-slice fetch --version 20.1.8 --triple <t> --libs … [--headers] [--cmake] -o ./out — download the
  standalone manifest, compute the closure, then pull only those members from the release zip via HTTP
  Range requests, plus optionally the whole include/ tree and lib/cmake/.

For the actual zip-over-HTTP extraction, prefer leaning on a maintained library rather than reimplementing
inflate: remotezip (PyPI) or unzip-http are fine. If you implement the range fetch directly, keep it
minimal: GET the last ~64 KB to read the End-Of-Central-Directory + central directory, locate each member's
local-header offset and compressed size, Range-GET those byte spans, strip the local header, and inflate.
Note in the README that GitHub release-asset downloads redirect to a CDN that supports byte ranges; the CLI
should verify with a HEAD/Accept-Ranges check and fall back to a full download with a warning if a
mirror ever doesn't.

Also write docs/usage-cmake.md showing how to consume an extracted slice (set CMAKE_PREFIX_PATH /
LLVM_DIR, or a raw link line built from resolve).

7. Workflows

.github/workflows/repackage.yml:

• Triggers: workflow_dispatch with inputs version (string, or latest), platforms (optional
  comma-separated triple filter), dry_run (bool, default true), draft_release (bool).
• Job discover: resolves the version, queries upstream assets via gh api, applies the optional filter,
  emits a JSON matrix of {triple, tarball_name, tarball_url, upstream_sha256} as an output.
• Job repackage (needs: discover, strategy.matrix from that JSON, runs-on: ubuntu-latest,
  fail-fast: false): for each triple, download → verify upstream sha256 (against LLVM's published
  checksum if available) → extract only needed paths → parse CMake → build manifest.json → zip → record
  sizes/sha256. Upload the zip + manifest as workflow artifacts. If dry_run, stop here.
• Job publish (needs: repackage, runs once): assemble index.json, create/ensure the v<version>
  release on this repo (respect draft_release), and upload all assets. Idempotent: GitHub rejects
  duplicate asset names, so delete-then-upload (gh release delete-asset / gh release upload --clobber).

Optional .github/workflows/watch-upstream.yml (schedule): check for new llvmorg-* releases and open an
issue or dispatch repackage.yml. Implement only after the core path works.

8. Disk and resource handling (hosted runner reality)

LLVM dev surfaces are multi-GB extracted. On ubuntu-latest:

• Stream-extract only the paths you need instead of unpacking everything:
  xz -dc clang+llvm-….tar.xz | tar -x --strip-components=1 -C work/ <needed-path-globs>
  (or tar -xJf … <paths> — either way you avoid writing the full tree).
• rm the downloaded tarball before zipping; clean work/ between steps.
• One platform per matrix job keeps each runner's disk bounded. If a particular platform still overflows,
  free space first (remove preinstalled toolchains) and document it; don't silently fail.

9. Correctness / edge cases to get right

• Windows: .lib extensions, llvm-config.exe, normalize zip entry paths to forward slashes.
• The config-specific exports file suffix (-release, -relwithdebinfo, etc.) varies — glob
  LLVMExports-*.cmake, don't assume -release.
• A target in LLVM_AVAILABLE_LIBS may have no IMPORTED_LOCATION (interface/header-only) → file: null,
  keep its deps.
• Distinguish internal LLVM deps (must resolve to a file in this zip) from external link flags (pass
  through as external). Don't drop external requirements — a consumer linking the closure needs them.
• Verify the upstream sha256 before processing; record both upstream and produced sha256 in the manifest.
• Make the zip reproducible-ish: stable file ordering, fixed timestamps if practical.

10. Repository layout

.
├── README.md                  # what/why, the seekability rationale, quickstart
├── .github/workflows/repackage.yml
├── .github/workflows/watch-upstream.yml   # optional
├── scripts/
│   ├── discover_platforms.py  # upstream asset discovery → matrix JSON
│   ├── repackage.py           # extract + parse CMake + build manifest + zip
│   └── build_index.py         # assemble index.json across platforms
├── client/
│   └── llvm_slice/…           # the consumer CLI
├── docs/
│   ├── usage-cmake.md
│   └── manifest-schema.md
└── tests/
    └── …                      # see acceptance criteria

11. Acceptance criteria

1. A workflow_dispatch with version=<a real recent LLVM release>, dry_run=true produces, for every
   discovered platform that has a dev surface, a *-dev.zip + manifest.json as workflow artifacts, and
   skips (with recorded reasons) any that don't.
2. manifest.json validates against docs/manifest-schema.md; for a spot-checked lib (e.g. LLVMOrcJIT),
   its deps match what llvm-config --link-static --libs orcjit reports for the host platform (use the
   host tarball to verify in a test).
3. client round-trip on a published (or locally served) zip: resolve for a small root set returns a
   closure, and fetch downloads strictly fewer bytes than the full zip (assert this) while producing
   files that pass a trivial link test against a tiny program using those libs on the host platform.
4. Re-running publish for the same version doesn't error on duplicate assets (idempotent clobber).
5. README.md clearly explains the tar.xz-vs-zip seekability rationale and the dependency-closure model,
   and credits that the build itself is upstream's (this repo only repackages).

12. Implementation order (spike first)

1. Spike before generalizing. Pick one smallish recent platform tarball, download it, and inspect the
   real structure: confirm lib/*.a, include/, lib/cmake/llvm/{LLVMConfig,LLVMExports,LLVMExports-*}.cmake,
   and bin/llvm-config actually exist and where. Adjust paths to reality before writing the matrix. Do not
   assume the layout — verify it.
2. Write repackage.py for that one platform end to end (extract → parse CMake → manifest → zip). Validate
   the manifest deps against llvm-config on that host tarball.
3. Write the client resolve/fetch against the local zip; assert the partial-fetch byte savings.
4. Write discover_platforms.py and wire the dynamic matrix; run dry_run=true across all platforms.
5. Add publish + index.json, make it idempotent, do a real (draft) release.
6. Docs, tests, then the optional upstream watcher.

Use gh for all GitHub operations. Commit in logical increments with clear messages. When something about
the upstream layout is ambiguous, inspect a real release rather than guessing.