chore: package local build outputs with nix#630
Open
kevinswiber wants to merge 35 commits into
Open
Conversation
Replace rustup in the Nix development shell with Fenix toolchains and expose reproducible Inspector, CLI, VSIX, and aggregate build outputs.
Match the build-only just build-all contract and ignore Nix result links.
Build the extension from its locked dependencies, bundle the raw CLI payload, and expose only the CLI and VSIX from the aggregate output.
Mark Git-less Nix packages with static nix-dev provenance while preserving the Cargo release version and version-document compatibility.
Keep Nix package tests enabled by making their Git-less fixtures and check-only tools explicit. Enforce the history cursor precondition in release builds so package checks use the normal release profile.
Define the test and production provenance helpers under separate cfg gates so Clippy does not see an unneeded return in test builds.
Use Nixpkgs' Nextest hook to parallelize package test binaries while retaining the sandboxed release-profile check.
Separate the fast CLI delivery package from the Git-less Nextest flake check while reusing a Crane dependency artifact across source changes.
Add ci-nix.yml as an experimental variant that runs alongside ci.yml without replacing it, while we evaluate a hermetic, store-cached gate. Each matrix leg (ubuntu, macos) runs `nix flake check` (the cli-nextest suite plus the devshell-tools drift check), `nix build .#build-all` (the store-backed CLI + Inspector + VSIX aggregate), and, on Linux only, the fmt + clippy gate via `nix develop -c just lint`. The Nix store is cached through GitHub's own Actions cache so crane's dependency artifacts are not recompiled every run. Windows stays in ci.yml: Nix has no native Windows support.
Add checks.clippy (cargoClippy, -D warnings) and checks.fmt (cargoFmt with the nightly formatter) so `nix flake check` is the full Rust gate — fmt, clippy, and the Git-less nextest suite — rather than tests only. clippy reuses the shared crane cargoArtifacts, and a dev-toolchain craneLib gives cargoFmt the nightly rustfmt that rustfmt.toml's unstable options require. Drop the now-redundant `nix develop -c just lint` step from ci-nix.yml; the flake check covers fmt and clippy directly.
Explore cross-compiling the test suite to Windows msvc on Linux/macOS via Nix and running the prebuilt cargo-nextest archive on a real Windows machine that needs no Rust build step. Add a windows-cross dev shell (Fenix stable host toolchain plus the msvc std for both shipped Windows targets, cargo-xwin, and the clang-cl / lld-link / llvm-lib cross tools), a windows-cross-archive recipe that emits the archive via `cargo-xwin env` + `cargo nextest archive`, and a report-only ci-nix-windows-spike workflow: ubuntu cross-builds the x64 archive, windows-latest runs it with no toolchain build. The lane is report-only because the suite resolves the test binary and fixtures through compile-time env!() paths that cargo-nextest's --workspace-remap cannot relocate across machines; a follow-up migrates those to runtime resolution.
…rchives
Replace the compile-time env!("CARGO_BIN_EXE_pointbreak"),
env!("CARGO_MANIFEST_DIR"), and env!("CARGO") — which bake the build
machine's paths into the test binaries — with runtime resolution that
prefers the vars Cargo and cargo-nextest set in a test's environment and
falls back to the compile-time value (or PATH for cargo). cargo-nextest
rewrites these for an archive built on one machine and run on another, so
the archived suite can locate the extracted binary and the
--workspace-remap'd fixtures on the target.
Add tests/support/env.rs (pointbreak_bin, manifest_dir, cargo_bin),
re-exported by the support module and included by standalone integration
tests; route the crate's #[cfg(test)] fixture lookups through a
pub(crate) test_fixtures::manifest_dir. Behavior is unchanged for
ordinary in-place runs, where the runtime and compile-time values are
identical (2929 host tests still pass). Cross-compiled to
aarch64-pc-windows-msvc and executed on real Windows, the archived suite
goes from 663 failures to green.
…igration The env!() → runtime migration landed, so the cross-compiled archive now runs green on real Windows (2924/2924, validated on an ARM64 VM). Update the spike workflow header and comments to reflect that, and note the lone build-on-target packaging test. The lane stays report-only until it is confirmed stably green on the x86_64 windows-latest runner.
The runtime-resolution migration skipped src/bench_support, so its
harnesses still resolved paths through the compile-time
env!("CARGO_MANIFEST_DIR"). Running the cross-compiled archive on a
Windows machine, those paths do not exist: four tests failed to resolve
the source root or a manifest, and the filesystem probe reported
"unavailable" instead of "ntfs" because it probed a nonexistent path.
Add a bench_support::manifest_dir helper — mirroring the cfg(test)-only
test_fixtures::manifest_dir, which these harnesses cannot use because
they also compile under the `bench` feature — and route the eight
run-time path uses through it. The fourteen
include_str!(concat!(env!(..), ..)) sites are deliberately unchanged:
they embed their bytes at compile time and never open a path at run time.
Crane defaults every derivation to the release profile, so the quality gates were paying optimized codegen for throwaway test binaries: the crate took 2m54s to compile and the suite 253s to run. Cargo's built-in `test` profile — what `just lint` and `just test` use locally, and what crane's own checks use — brings that to 25s and 179s on the same machine. Add a matching test-profile dependency set so clippy and nextest share artifacts; the delivery build keeps its own release artifacts. Run clippy on Linux only, mirroring ci.yml. Its result is platform-independent and `--all-targets` compiles the whole test surface, so running it on both platforms bought a second full compile for nothing. Replace `nix flake check` in CI with the individual checks realised in order (fmt, clippy, test, build). flake check batches every check in arbitrary order, so it cannot express a cheap gate that returns early; naming them in sequence can. Expose devshell-tools as a package so CI can realise it without hardcoding a system string per runner, and drop the blanket `-L` in favour of dumping the failing build log on failure.
The shore -> pointbreak rename landed in 0.7.0, so the guards that policed it are no longer describing a live invariant: the 0.7.0 hard-cutover narrative, the legacy-word scanner and its classification table, and the Hunk-era branding checks. Remove those seven tests and the helper machinery behind them. This also shrinks what the Rust build depends on. The retired every_retained_public_legacy_reference guard walked docs/, skills/, scripts/, benches/, and the fixture trees wholesale, so every file in them was an input to the test binaries. The durable identity contracts stay: crate and command identity, the canonical organization repository, release badges, Cargo and VS Code packaging metadata, the skills install route, and the resolved-store doc guidance.
cargoClippy installed its compiled target directory as the derivation output: a ~250MB archive that nothing consumes, changed on every commit, and cost the build a 632MB -> 164MB compression pass to produce. It also dominated the CI store cache, where it was the one large entry with no reuse value between commits. Set doInstallCargoArtifacts = false, matching the nextest gate, which already produced an empty output for the same reason. The reusable part is the shared dependency artifacts, which are built separately. The clippy output is now 0B.
cache-nix-action tarred the whole /nix/store into a single Actions cache entry: 4.87GB on Linux and 4.46GB on macOS, mostly paths cache.nixos.org already serves for free. Together they exceeded the 10GB repository limit on their own, so the two evicted each other and Linux ran with a cold store on every run, rebuilding the toolchain and every dependency. Magic Nix Cache stores individual store paths over the same GitHub Actions cache and skips anything fetchable from the upstream cache, so only this project's own outputs are kept — a few hundred megabytes rather than a whole-store snapshot. It still needs no external account. Disable its FlakeHub upload, which is a separate account-backed service this lane does not use.
The job named "Format and lint" ran `just nix-check`, whose `nix flake check` builds every check — clippy, the full nextest suite, tool drift — with no store cache. It took 14-18 minutes and, since ci-nix.yml realises those same derivations on every pull request, it was running the entire suite a second time in parallel from cold. Split the Nix-file linting (nixfmt, statix, deadnix) into `just nix-lint` and point the workflow at it: it builds nothing and finishes in seconds. `just nix-check` keeps the previous behaviour for local use. `nix flake check --no-build` would not work as a cheap substitute here: cleanCargoSource filters a derivation output, so evaluation has to build the source derivation first.
CI resolved its Rust toolchain through rustup while CONTRIBUTING.md tells contributors to set up with `nix develop`, so the compiler that gated a change was not the one a contributor built with. Point the Linux and macOS legs at the flake instead: `nix develop .#ci -c just <recipe>` runs the existing recipes against the toolchain pinned by flake.lock, keeping rust-cache and the Windows sharding that make this lane fast. Windows still uses rustup, because Nix has no native Windows support; retiring that needs the cross-compiled nextest archive to graduate out of its report-only phase. Add a lean `.#ci` dev shell for this. The default shell carries the interactive extras — cocogitto, which is built from source here, plus gh and cargo-edit — that a workflow never uses. Reduce the parallel Nix workflow to what only it can check. It was re-running clippy and the whole suite on the same code through the same toolchain, which cost roughly 1.5x for a duplicate answer. What is unique is that the flake still evaluates, the dev shell still carries its pinned tools, and a hermetic no-network build of the shipped artifact still succeeds. The sandbox-only regressions that drops — a test needing a tool the derivation does not provide — move to a nightly full flake check.
Reinstate the phased pure-Nix gate on Linux for every pull request. Both lanes now share the flake-pinned toolchain, so this is not re-checking lints ci.yml already runs; what it establishes is that the suite passes in a sandbox, with no host tools, no network, and no `target/` restored from a cache. ci.yml gets its speed from that restored target, which makes a stale-artifact false green possible there and impossible here. macOS stays nightly rather than per pull request: it measured 22m37s against Linux's 15m46s, which is not worth adding to every change for a platform ci.yml already covers. The nightly job runs `nix flake check` rather than the named phases, so a check added to the flake cannot be silently left ungated by the explicit phase list.
This session replaced the toolchain source, the store cache, the build profile, and the division of labour between the two lanes, each on the back of a measurement that is not visible in the resulting YAML. Several of the choices read as mistakes without it: a test profile where release is expected, a clippy gate that installs nothing while nextest does, a lint job that no longer runs the checks, an unusual cache action, and no rust-cache on the Nix lane. Add docs/ci-architecture.md with the lane split, the numbers behind each decision, the alternatives that were measured and rejected, and the conditions that should reopen them. Point the workflows and the flake at the relevant section from the place the decision is expressed, so a future editor meets the rationale before undoing it.
Magic Nix Cache stored one Actions cache entry per store path — 1788 of them here — and the resulting API call volume tripped GitHub's rate limiter. On throttle it logs "Not trying to use it again on this run" and disables itself, so the job silently finishes against a cold store while still paying to build everything. That is the opposite ceiling from the approach before it: cache-nix-action tarred the whole store into one entry per platform, 4.87GB on Linux plus 4.46GB on macOS against a 10GB storage limit, so the two evicted each other. Hestia packs paths into content-defined chunks — few large entries — which is the point between the two. It needs no account either; uploads authenticate with the runner-injected ACTIONS_RUNTIME_TOKEN. Pin it by commit SHA, as this repository already pins its other third-party actions, and record all three approaches and their failure modes in the CI architecture notes.
kevinswiber
marked this pull request as draft
July 25, 2026 02:32
A cold run drains 2.3 GiB as 491 paths packed into 37 cache entries, roughly 48x fewer objects than one entry per store path, and no throttling appears in the log. Re-running an unchanged commit costs 48s because every derivation output is then a cache hit. Note the caveat with it: that figure is the identical-input case. A real change still recompiles the crate and its test binaries, so a normal pull request lands between the cold and warm numbers.
The Nix lane's store cache shares GitHub's 10GB per-repository quota with ci.yml's rust-cache entries, and nothing was reclaiming the Nix side. Without collection the default-branch scope grows until GitHub starts evicting, which is how the previous cache approach ended up cold on every run. Add the nightly collection upstream recommends, adapted to use the released binary through the same pinned action the writing lanes use rather than building hestia from source. It only considers paths hestia itself pushed: liveness comes from its own per-branch-and-system roots, so entries written by rust-cache, setup-node, or the Nix installer are never candidates. This is the only workflow holding actions:write, which the REST cache deletes require, and it carries a dry-run input for inspecting a plan before trusting it.
The spike ran the archived suite as a single Windows leg, so the lane cost 15m20s — cross-compile then run, serialized — against 10m02s for ci.yml's sharded rustup legs. Being slower than what it aims to replace was the reason it could not graduate. Fan the run across three shards with the same slice:N/3 partitioning ci.yml uses. The split is even and complete: 2922 tests, 974 per shard, reconciling exactly. The shape here is better than ci.yml's rather than merely equal. There, each Windows shard rebuilds the test binaries, so sharding multiplies compile work. Here the archive is cross-built once on Linux and all three shards consume it, so sharding buys execution time and nothing else.
The test existed to prove the shore -> pointbreak rename had taken: most of its body asserted that no `shore` or `shore.exe` landed in the install root. That rename shipped in 0.7.0. The one durable claim it still made, that installing yields exactly one binary named pointbreak, is already asserted from cargo metadata by package_identity_declares_only_pointbreak_binary — cargo install places exactly the [[bin]] targets metadata enumerates. Retiring it also clears the last on-target Rust build out of the Windows cross-archive lane, which had made whichever shard drew this test the long pole at 5m46s against 2m26s for the fastest. The no-build premise there now holds without exception. Worth knowing what this gives up: nothing now runs `cargo install`, which is the route the README and installation guide give users. The release workflows build and verify the published binaries, but that specific path is no longer covered directly.
The spike is green on x64 and now cheaper than what it replaces: sharded three ways it finished in 9m57s against 10m02s for the rustup legs, and retiring the on-target cargo install test removed the last thing any shard had to compile. Fold it into ci.yml as the Windows test path and delete the separate spike workflow. Windows test legs no longer install a Rust toolchain at all: the binaries are cross-compiled once on Linux with cargo-xwin and the shards only execute them. Rustup does remain on Windows for three other legs — the --all-features type-check, store qualification, and git parity — so this frees the test execution, not the platform. Also drop the Linux test leg, which ci-nix.yml's sandboxed gate already covers, and keep macOS as a plain cargo run so it retains an incremental target/. The Rust gate is now split by platform: Linux hermetic, macOS incremental, Windows cross-compiled. Keep test-windows-check on rustup deliberately. `just check-types` is --all-features while the archive is built with default features, so the cfg(windows) arms behind bench and gix-parity are compiled for Windows nowhere else.
The windows-shards rust-cache key dated from when three Windows legs compiled the identical test-binary set and wanted one entry between them. Those legs now execute a cross-compiled archive and have no target/ to cache, so the key described a sharing relationship that no longer exists.
The harness existed to measure levers on the real Windows runner image one at a time, and that work concluded — the sharding it informed has since been superseded anyway. Its shard probe measured rust-cache warmth across three Windows legs that now execute a cross-compiled archive and have no target/ to warm. It was dispatch-only and referenced by nothing, so removing it changes no gate. Git history keeps it if the runner ever needs measuring again.
kevinswiber
marked this pull request as ready for review
July 25, 2026 04:52
Reintroducing a compile-time env!("CARGO_MANIFEST_DIR") or
env!("CARGO_BIN_EXE_pointbreak") is easy to do by accident and fails
nowhere except the Windows shards, where the cross-compiled archive is
relocated and the baked path points at the Linux build machine. The
failure reads "The system cannot find the path specified" and names no
cause, which cost a long debugging round the first time.
Fail on Linux instead, naming the file, the line, and the resolver to
reach for, and pointing at the CI architecture notes for why the rule
exists. include_str!/include_bytes! are exempt, scoped to the enclosing
statement so an unrelated embed nearby cannot excuse a real path.
The guard found one site the original migration missed:
examples/support/review_example_pack.rs resolved the manifest directory
at compile time to run git in it, and tests/review_example_pack.rs
includes that file, so it was already compiled into a Windows test
binary — latent rather than failing only because nothing exercised it
there.
`nix build` prints nothing on success, so the Linux gate finished green with no test counts anywhere in its log — there was no way to tell from a run whether the suite had executed. It had: the step takes minutes, and a substituted result would take seconds, since the nextest derivation installs a 0-byte output. But "it must have, look at the duration" is a poor substitute for saying so. Recover the counts from the derivation's own build log instead of restoring -L, which streams every compile line and is what made these runs unreadable. Nextest colourises even in the sandbox, so the escape codes come out before matching. On a cache hit there is no local log, and the step says that plainly rather than printing a blank. That case is not a coverage gap: the derivation output is the proof those inputs passed, from an earlier run.
ci.yml showed a macOS leg and Windows legs and no Linux one, which reads as a hole in the gate. Linux was covered the whole time, by a job in a separate workflow, but nothing about the checks list said so. Move it in and name it for the platform — test (ubuntu-latest), beside test (macos-latest) and test (windows-latest N/3) — so the matrix is legible from the checks list alone. The separate workflow made sense while the Nix lane was an experiment running beside the real gate; it is the real gate now. What is left runs nightly and on demand only: the hermetic check on macOS, too slow to gate at 22m37s against Linux's 15m46s, and `nix flake check` in full, which covers any check the phase list in ci.yml does not name. Drop its push and pull_request triggers, and the job-level guard they made necessary — a job that only skips is noise in the checks list.
The Windows cross-archive jobs used upload-artifact@v4 and download-artifact@v4, which run on Node 20 and warn that the runtime is deprecated. The rest of the repository was already on v7 and v8; these two were left behind when the jobs were written. An audit of every action reference found nothing else on Node 20. checkout@v6, setup-node@v6, setup-uv@v8.1.0, rust-cache@v2, and action-gh-release@v3 all declare node24 despite newer majors existing, so bumping them buys no runtime currency. The three SHA-pinned actions are current too: create-github-app-token resolves to v3.2.0, hestia to v2.0.0, and action-actionlint runs on docker, where the deprecation does not apply.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
nix develop --no-write-lock-file --command just build-allnix build .#inspector --no-link --print-build-logsnix build .#cli --no-link --print-build-logsnix build .#vscode --print-build-logsnix build .#build-all --no-linkcargo +stable test --test build_provenancecargo +nightly fmt --all -- --checknix flake check --no-build --no-write-lock-filegit diff --check