feat(sha256): platform-neutral SHA-256, unblocking the XS daemon bundle

[kriscendobot]

Implements the approved design designs/platform-neutral-hash.md (design PR #824).

What this unblocks

node packages/daemon/scripts/bundle-bus-daemon-rust-xs.mjs failed on llm with 16 unresolvable node: imports, so daemon_bootstrap.js was never produced, which blocks test:rust and full endor daemon integration (#600). It now produces the bundle.

@endo/sha256

A small buffer-oriented package: sha256(bytes) -> Uint8Array(32) and sha256Into(out, bytes, offset). Synchronous, raw digest bytes in and out, TypeError / RangeError rather than coercion. Conditional exports:

condition	build	backing
node	src/sha256-node.js	node:crypto createHash('sha256')
xs	src/sha256-xs.js	the Endor Rust host's hostSha256Init / hostSha256UpdateBytes / hostSha256Finish
browser, default	src/sha256-browser.js	pure-JS synchronous SHA-256

The XS build composes the host's existing binary-safe incremental triple, so this increment needs no new Rust. It never uses the host's one-shot hostSha256, which takes a UTF-8 string and so corrupts any input byte above 0x7f. It prefers a one-shot hostSha256Bytes if the host ever grows one (the design's deferred Rust optimization), and falls back to the pure-JS digest under an XS engine that is not the Endor host.

default maps to the pure-JS build so the package is safe under any bundler that sets none of the three conditions.

Open question, resolved as the design proposed. WebCrypto's crypto.subtle.digest is async and cannot back a synchronous sha256(bytes) -> bytes; the only in-graph consumer (makeBlobRefExo) content-addresses inside a synchronous exo factory. So the browser build is pure JS, and a separately named async sha256Async is left as a future addition rather than built speculatively. Say the word if you want WebCrypto in the browser build instead, which would force the whole BlobRef construction path async.

The rest of the bundle blockers

@endo/sha256 cleared one of the sixteen. Two more legs, both the same shape, cleared the other fifteen:

• @endo/exo-git reached readOnly and wrapBackend through the @endo/platform/fs/extended index, which also re-exports makeNodeFilesystem / makeNodeFsBackend and so dragged in node:fs, node:fs/promises, node:path. It now imports those two modules directly.
• @endo/git and @endo/host-spawner were statically imported by manager.js / host.js. Both shell out to a host process and could not run under XS regardless, so they are now injected as DaemonicPowers.hostTools, exactly as better-sqlite3 already reaches the daemon as an injected Database. Node supervisors supply them from host-tool-powers-node.js; the XS supervisor supplies none and gets stand-ins that refuse with a diagnosis, so a git or shell formula under XS reports what is unavailable rather than failing to link.

The bundler now passes the xs condition, which is what the repository's own bundler comments prescribe for steering retention. Without it @endo/sha256 would silently take its default arm (pure JS) instead of binding to the host's Rust-native digest.

Verification

All run locally on this branch.

• XS daemon bundle generates. node packages/daemon/scripts/bundle-bus-daemon-rust-xs.mjs -> Wrote rust/endo/xsnap/src/daemon_bootstrap.js (2136164 bytes), down from a 16-failure TypeError. node --check on the output passes. The bundle's hostSha256Init occurrence count rises from 3 (all in bus-manager-rust-xs-powers.js) to 6, which is the evidence that the xs condition selected sha256-xs.js rather than the pure-JS default arm.

• @endo/sha256: 72 tests pass. All three builds cross-checked against node:crypto over FIPS 180-4 vectors and lengths 0/1/55/56/57/63/64/65/119/120/128/1000; bytes above 0x7f; views into a larger buffer; sha256Into offset, BYOB-into-a-view, and rejection cases. fast-check properties cover the cross-build-agreement claim over generated input (seeded and shrinking, replacing unseeded randomBytes). The xs build is exercised against a stand-in host implementing the Rust contract: the one-shot preference, the no-host fallback, a malformed hex digest, one-shot returns that are a short buffer / long buffer / number / string, and a throwing update that must not leak a host handle.

• @endo/platform: 309 tests pass (blob-ref, local-blob, snapshot-hash included). @endo/chat: 800 pass. @endo/exo-git: 3 pass.

• @endo/daemon: the suite does not run clean in this sandbox, so it was run against a baseline of the same tree with these changes stashed. test/git.test.js: 6 failures before and after, identical (host git config and /tmp noexec). test/shell.test.js + test/mount-snapshot-and-entry.test.js + test/store-controller.test.js: 1 failure before and after, identical (spawn ... EACCES, /tmp noexec). No regression, and the passing git and shell tests exercise the new injected hostTools path end to end.

• Lint and types: eslint reports 0 errors across packages/{sha256,daemon,platform,exo-git,chat}; tsc is clean in each of those five; prettier --check clean. yarn build:types:gen re-ran, so the composite tsconfigs are in sync.

• TypeDoc: node node_modules/typedoc/bin/typedoc at the root exits 0 with no errors. That gate reddened once on this branch, because it type-checks JavaScript that packages/daemon's own tsconfig does not (checkJs: false), so the new daemon test passed the package's lint:types and still broke test. Fixed in 80521b46c; the local reproduction above is what closes that parity gap.

Not run here: yarn --cwd packages/daemon test:rust needs a built target/release/endor, and the full repository suite.

Follow-up for #600

Once this merges, #600 needs exactly one thing: rebase on llm and re-run yarn --cwd packages/daemon test:rust. The bundle generator it was blocked on now succeeds, and daemon_bootstrap.js is gitignored and generated, so no artifact needs to move between the branches. No source change is required in #600 for the crypto blocker.

Deliberately left for a separate change, as the design says: migrating native-git-backend.js's two createHash sites to @endo/sha256 (consistency only, it leaves the XS bundle by exclusion), and the Rust one-shot host_sha256_bytes (this build already prefers it if it appears).

Panel rounds

The scripted jury panel reviewed this twice and both rounds found real defects, fixed in 1ae5f99a1 and 75e2d35fb.

Round 1 (at 3a7f81bac): a wrong-sized or non-buffer one-shot XS host return became a silent wrong content address; the streaming triple leaked a host handle when an update threw; the pure-JS padded length used a signed 32-bit shift that diverges from the node build above 2 GiB. It also blocked on missing changesets and on rust/endo/README.md still describing this bundle as failing while the new code cited it as the durable record.

Round 2 (at f3499b3f3): three seats independently ran the same attack. assertBytes used instanceof, which a Proxy over a real Uint8Array satisfies, and the pure-JS build read bytes.length four separate times, so a proxy answering different lengths returned the empty-input digest for eight bytes of content while the node build threw on the same value. It now brand-checks through %TypedArray%.prototype's length accessor and reads the length once. A detached buffer likewise diverged (node: empty digest, pure JS: V8's own throw) and now digests as empty everywhere, matching node. The hostSha256Bytes digest was aliased rather than copied, and the no-host fallback memoized so an early digest could pin pure JS for the process; both fixed.

The prover seat broke three things that reddened nothing and each now has a test: makeSha256Into's digest guard, every conditional-export arm (it pointed xs at a typo and the whole suite still passed), and provideHostToolPowers' refusal diagnostics. CI now also generates the daemon bundle, which was ungated because the artifact is gitignored.

Remaining juror suggestions are preferences and follow-ups rather than defects, and are listed in the panel record rather than acted on here.

Draft for review.

[kriscendobot]

Ready for review at 80521b4, all 25 CI checks green.

What it does. Implements the approved designs/platform-neutral-hash.md: @endo/sha256 with node / xs / browser+default conditional builds, BlobRef content-addressing through it, and the chat shim folded into it. The XS daemon bundle generates again, which is what #600 was blocked on.

What the survey did not anticipate. @endo/sha256 cleared one of sixteen unresolvable node: imports. @endo/exo-git reached readOnly and wrapBackend through the @endo/platform/fs/extended index, dragging node:fs onto the graph; and the @endo/git exclusion the design deferred turned out to be a prerequisite, together with @endo/host-spawner. Both now reach the daemon core as an injected DaemonicPowers.hostTools, the way better-sqlite3 already did.

One open question for you. The design proposed pure-JS (sync) as the browser build with WebCrypto as a separately named future async export, rather than forcing BlobRef construction async. That is what shipped. Say the word if you want WebCrypto in the browser build instead.

The panel found three real defects, all fixed and each now covered by a test: a wrong-sized or non-buffer one-shot XS host return became a silent wrong content address; assertBytes used instanceof, so a Proxy answering different lengths returned the empty-input digest for eight bytes of content while the node build threw; and a detached buffer diverged the same way. Details are in the PR body and the commit messages.

Deferred, as the design says: migrating native-git-backend.js's two createHash sites (consistency only now that @endo/git leaves the bundle by exclusion), and the Rust one-shot host_sha256_bytes (the XS build already prefers it when present, with the branch tested).

_{harness claude · garden be985a3c}

[kriskowal]

Please address the one caveat, rebase, retcon, weave, and merge.

[kriskowal]

We should rename this condition endor.

[kriscendobot]

Follow-up work is complete at head f6cd2b4fab0b1bbde601ea7a05576cb8e7af76bd.

• 67b6b015d8: migrated the native git backend’s two SHA-256 sites to @endo/sha256, and added the binary one-shot XS host function with its hostSha256Bytes alias, ambient declaration, append-only callback registration, and raw-byte tests.
• f6cd2b4fab: updated the lockfile for the @endo/git workspace dependencies.
• Declined items: none.

Verification: @endo/git typecheck passed; its AVA suite passed (3 tests); ESLint completed with 0 errors (9 existing warnings); @endo/daemon typecheck passed; all three Rust/XS bundles regenerated successfully; cargo test --manifest-path rust/endo/xsnap/Cargo.toml crypto_sha256 passed (1 test, 119 filtered); and yarn install --immutable passed.