Skip to content

Fix offline peer dependency resolution by synthesizing bun's manifest cache - #103

Draft
typedrat wants to merge 13 commits into
nix-community:masterfrom
synapdeck:typedrat/offline-manifest-cache
Draft

Fix offline peer dependency resolution by synthesizing bun's manifest cache#103
typedrat wants to merge 13 commits into
nix-community:masterfrom
synapdeck:typedrat/offline-manifest-cache

Conversation

@typedrat

Copy link
Copy Markdown
Contributor

Fixes #71 (and the offline-install portion of #77).

Note

Stacked on #101 and #102 — since PRs from a fork can't target another fork branch, this branch includes their commits; only the last nine commits are new here. I'll rebase and mark it ready once those two land.

bun needs npm package manifests to do peer dependency resolution, even with --frozen-lockfile. When a dependency graph has peers in it, the lockfile resolution comes back invalid and bun goes looking for the manifest; if it isn't in the cache, it downloads it. Our cache only ever contained the extracted packages and no *.npm manifest files, so inside the sandbox that download fails with ConnectionRefused downloading package manifest. That's why adding a peer-heavy package (the reporter's SvelteKit starter) broke builds of unrelated workspace packages while packages with no peers anywhere kept working: bun never reads a manifest for those.

There's no flag to turn this off (--offline isn't a real bun flag), and fetching the real registry documents in a fixed-output derivation isn't reproducible since they change whenever a new version is published upstream. So instead we generate bun's binary manifest cache ourselves:

  • All the fields bun reads out of a manifest (dependency ranges, optional peers, bin, os/cpu) are already sitting in each bun.lock entry's inline metadata, so bun2nix now reconstructs a per-version manifest from the lockfile and writes it as a manifest attr on the entry in bun.nix. No network involved, which also means it works the same in the wasm CLI.
  • At build time fetchBunDeps feeds those attrs to a new manifest subcommand of cache-entry-creator, which writes real bun-npm-manifest-cache-v0.0.7 files into the cache. The install hook now exports BUN_MANIFEST_CACHE=2 (bun ignores the on-disk cache otherwise), and the manifests get public_max_age = u32::MAX so they can't expire mid-build.
  • cache-entry-creator is rewritten from Zig to Rust along the way. The manifest serializer, wyhash, and cache folder naming all have to match bun bit-for-bit on both the generation and build side, and maintaining that port twice — once in Rust for bun2nix, once in Zig — seemed like a bad time. It all lives in a shared bun2nix-core crate now; the symlink subcommand keeps the same CLI as the Zig tool and its test vectors came along as Rust tests.

Non-default registries work too. bun keys those manifests by a hash of the registry URL from its config, not from the lockfile, so bun2nix parses the project-local bunfig.toml/.npmrc next to the lockfile (npmrc wins per key, same as bun) and names the .npm files accordingly. Global config, env vars, and --registry are ignored on purpose: the sandboxed bun can't see any of them, so a registry configured only there could never resolve offline anyway, and reading them would bake machine-specific state (and auth tokens) into bun.nix.

Pre-release versions work too. bun looks a version up in the manifest's releases or prereleases map depending on whether it has a tag, and tag equality compares a hash of the pre-release span, so 3.0.1-alpha.1 has to be stored in the right map with the right hash — an earlier revision stripped the tag and stored it as 3.0.1, which bun can never find. The version parser now follows bun's Tag.parse state machine, and the os/cpu fields go through a port of bun's Negatable handling ("none", "!win32", bare strings) since bun writes those forms back into bun.lock.

Getting the full opencode repro from #77 green surfaced one more offline killer fixed here: with minimumReleaseAge configured (opencode's bunfig sets it), bun silently rejects any cached manifest that doesn't claim extended data — it demotes the entry to expired and refetches. Synthesized manifests now set has_extended_manifest; their zeroed publish timestamps read as "published at epoch", which passes every age gate, the right answer for lockfile-pinned versions. (Two hook-level killers found in the same investigation — lockfile drift forcing re-resolution, and the catalog: rewrite mangling non-registry specs — apply to master independently of the manifest work and live in #102.)

Testing: two new flake checks install fixtures with bun.lock excluded from the source, which forces bun through the manifest-cache path — without the synthesized manifests they fail with exactly the error from #71. One covers the default registry, the other uses a real alternate registry (registry.npmmirror.com). On top of that there are golden tests pinning the wyhash values and header layout, round-trip tests for the binary format, and a tripwire that fails if bun ever bumps the manifest format version, with notes on how to re-port. As an end-to-end test, the full opencode workspace — ~3,200 packages: workspaces, 330 catalog references, 15 patched deps, pre-release pins, a github dep, a pkg.pr.new tarball, vendored tarballs — installs completely offline from a cache generated by this branch, once its lockfile is refreshed to clear the drift caught by #102 (as committed upstream it fails with that PR's diagnostic instead).

Compatibility: existing bun.nix files build unchanged (no manifest attrs → empty manifest cache → old behavior); you get the fix by regenerating. The wasm export grew two optional trailing args for the config file contents. The obvious risk is the coupling to bun's manifest ABI, but it's been stable at v0.0.7 across 1.3.x, and the tripwire turns a future bump into a loud test failure instead of silently broken installs.

Not covered here: auth for private-registry tarball fetching (the cache keys are already right, since credentials aren't part of the hashed URL, but authenticating the FODs is a separate problem), and there's no end-to-end fixture for a path-style registry URL — that's unit-tested only, since I couldn't find a public registry shaped like that to commit a fixture against.

typedrat added 13 commits July 25, 2026 17:51
bun does not keep entry kinds and tuple arities in one-to-one
correspondence: github entries can carry an integrity hash (arity 4) and
remote or vendored tarball entries can carry inline metadata (arity 3), so
dispatching on arity misroutes both — a github dep lands in the npm parser
and a pkg.pr.new tarball lands in the git parser, failing with
MissingGitRef.

Dispatch on the identifier's resolution instead (github:/git+/http(s)://,
npm for bare versions at arity 4, file paths otherwise). The identifier is
split at the '@' that ends the package name — the second '@' for scoped
names — which also fixes resolutions that contain '@' themselves, like
https://pkg.pr.new/@scope/pkg@sha tarball URLs. Vendored tarball paths with
no file:/./ prefix (e.g. "vendor/pkg-1.0.0.tgz") are now accepted as file
packages.
A "<workspace-name>/<pkg>" lockfile entry records its file-dependency path
relative to that workspace's directory, but bun.nix paths resolve from the
project root — so a vendored tarball under packages/app rendered as
./vendor/pkg.tgz (missing) and a sibling workspace's ../app/vendor/pkg.tgz
escaped the root entirely.

Factor package building out of convert_lockfile_to_nix_expression into
build_packages, and rewrite CopyToStore paths for entries nested under a
workspace (longest workspace-name prefix wins) with lexical ././..
normalization.
…network

Two hook fixes for failure modes found installing opencode offline. Both
stem from the same bun behavior: any drift between package.json and
bun.lock makes bun re-resolve the affected dependencies, and re-resolving a
git/github/remote-tarball dependency downloads it unconditionally — fatal
in the sandbox no matter how complete the cache is.

- Detect trustedDependencies / patchedDependencies drift between the root
  package.json and bun.lock and fail with an actionable message (refresh
  bun.lock, regenerate bun.nix). Projects routinely commit a lockfile whose
  copies of these sections lag package.json (opencode's does); without the
  check the drift surfaces as a wall of ConnectionRefused errors at resolve
  time with no hint of the cause.
- Leave catalog: references pointing at non-registry specs (github:, git+,
  tarball URLs, file:) unrewritten. bun resolves those natively from the
  lockfile's catalog section; rewriting them to their resolution registered
  as a changed spec and forced a re-resolve.

The hook now runs the prep script whenever bun.lock exists (previously only
when it contained catalog: refs), so the drift check covers every project.
Runs resolve-catalog.ts against three inline fixtures: drifted
trustedDependencies/patchedDependencies must fail with a diagnostic naming
the drifted entries, identical-but-reordered sections must pass, and
catalog: refs must be rewritten to exact versions except when they resolve
to non-registry specs. The drifted fixture has no catalog: refs, pinning
that the check runs for plain projects too.
…eator

Introduce a programs/ cargo workspace with a shared bun2nix-core crate:
vendored bun Wyhash11, the bun cache-folder-name port, and verbatim
structs + serializer + multi-version builder for bun's binary .npm
manifest cache format (bun-npm-manifest-cache-v0.0.7). Rewrite
cache-entry-creator from Zig to Rust on top of it, keeping the symlink
behavior and adding a manifest mode that turns EntryMeta JSON into .npm
cache files. Nix package builds updated to build from the workspace.
The dependency-graph metadata bun stores inline in each npm lockfile
entry mirrors the abbreviated registry manifest, so the deserializer
rebuilds a per-version VersionMeta from it with no network access.
Default-registry entries render a manifest attr in bun.nix (tarball URL,
dependency groups, bin/os/cpu, install-script flag) for the offline
manifest cache to consume downstream.
Collect EntryMeta records from bun.nix manifest attrs and run
cache_entry_creator manifest to write the .npm files bun consults at
resolve time, merged into the dependency cache. The install hook exports
BUN_MANIFEST_CACHE=2 so bun reads the on-disk manifest cache instead of
hitting the network. Manifest-less bun.nix files yield an empty cache
and build as before.
…nity#71)

react-dom@19 peer-depends on react; bun.lock is excluded from the
fixture source, so bun must resolve peer deps from the synthesized
manifest cache — the code path that previously fell back to the network
and failed in the sandbox. flake.nix now skips auto-importing fixture
bun.nix files (they are package expressions, not flake-parts modules).
Also pins the manifest format version as a tripwire.
…ity#71)

Key each package's .npm manifest by the registry bun will compute at
offline-install time. A new bun2nix-core config module parses
project-local bunfig.toml/.npmrc contents (.npmrc overrides per key)
and ports bun's scope_for_package_name; global/env/CLI layers are
deliberately ignored because the Nix sandbox's bun cannot see them
either. Non-default entries carry a registry attr inside their bun.nix
manifest block, cache-entry-creator groups by (name, registry) and
writes <wyhash(name)>-<wyhash(registry)>.npm with the registry's
url_hash in the header, and the wasm CLI passes the two config files'
contents across the boundary (no filesystem access in core).
…ure)

End-to-end guard: the fixture pins react/react-dom to
https://registry.npmmirror.com via a committed project-local
bunfig.toml; bun.lock is excluded from the source, so sandbox bun must
find the registry-keyed <hex>-<hex>.npm manifests under the url_hash it
derives from that same config — proving the generation-time key matches
byte-for-byte.
…os/cpu

Pre-release versions were stored tag-stripped in the releases map, but bun's
find_by_version only searches prereleases when the queried version has a tag,
so any pre-release pin (e.g. nitro@3.0.1-alpha.1) missed the synthesized
manifest and fell back to the network. Parse the full
major.minor.patch[-pre][+build] form with a port of bun's Tag.parse state
machine, intern pre/build tags (Tag.eql compares wyhash11 of the pre span),
and split versions into the releases/prereleases maps, each sorted ascending
with bun's numeric-aware order_pre (find_best_version scans from the end).

os/cpu parsing now ports bun's Negatable accumulator: "any", "none",
"!name" negation, and unrecognized-value handling, all of which bun writes
back into bun.lock metadata.
With minimumReleaseAge configured (e.g. opencode's bunfig.toml), bun's
manifest cache load rejects any manifest whose has_extended_manifest flag
is unset — it demotes the entry to expired and schedules a network refetch,
which is fatal in the sandbox. Set the flag: our zeroed
publish_timestamp_ms values read as "published at epoch", which passes
every age gate — the right answer for lockfile-pinned versions.
@TheRealGramdalf

Copy link
Copy Markdown

I can confirm that this PR works "in the wild" on a monorepo I'm working with at https://github.com/TheRealGramdalf/mindwtr-flake. This fixes a build failure and makes my flake possible, so I'd love to see this merged. Happy to provide more info if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workspace dependency install fails in nix build when adding additional workspace package

2 participants