fix(daemon): restore XS bundle generation on llm - #937
Open
kriscendobot wants to merge 10 commits into
Open
Conversation
`bundle-bus-daemon-rust-xs.mjs` failed with sixteen unresolvable `node:` imports. `@endo/sha256` cleared one of them; this clears the other fifteen, so `daemon_bootstrap.js` is produced once more and `test:rust` and the endor daemon integration are unblocked. Two legs, both "a module on the graph statically imports something XS cannot resolve": - `@endo/exo-git` reached `readOnly` and `wrapBackend` through the `@endo/platform/fs/extended` index, which also re-exports `makeNodeFilesystem` and `makeNodeFsBackend` and so pulled in `node:fs`, `node:fs/promises`, and `node:path`. It now imports the two modules it uses directly. - `manager.js` and `host.js` statically imported `@endo/git` and `@endo/host-spawner`, which between them reach nine `node:` builtins and could not run under XS regardless: both shell out to a host process. They are now injected as `DaemonicPowers.hostTools`, the way `better-sqlite3` already reaches the daemon as an injected `Database`. Every Node supervisor supplies them from `host-tool-powers-node.js`; the XS supervisor supplies none and gets stand-ins that refuse with a diagnosis, so a `git` or `shell` formula under XS reports what is unavailable instead of failing to link. The bundler now also passes the `xs` condition, which is how the repository's own bundler comments say to steer retention. Without it `@endo/sha256` would silently resolve its `default` arm, a pure-JS digest, rather than binding to the host's Rust-native SHA-256. Design: designs/platform-neutral-hash.md Refs: rust/endo/README.md, "The XS daemon bundle pulls in Node-only packages" Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A module that statically imports `node:crypto` cannot pass through the
SES/XS bundler, which has no `node:crypto` to resolve. `@endo/sha256`
gives such a module one import specifier that resolves to a working
digest on every target, through conditional exports:
- `node` -> `node:crypto`'s `createHash('sha256')`, copied off the
`Buffer` so the result is a plain `Uint8Array`.
- `xs` -> the Endor Rust host's binary-safe incremental triple
(`hostSha256Init` / `hostSha256UpdateBytes` / `hostSha256Finish`), so
the first increment needs no new Rust. It prefers a one-shot
`hostSha256Bytes` if the host ever grows one, and falls back to the
pure-JS digest under an XS engine that is not the Endor host. It
never uses the host's one-shot `hostSha256`, which takes a UTF-8
string and so corrupts any input byte above 0x7f.
- `browser` and `default` -> a pure-JS synchronous SHA-256, lifted from
the browser stand-in in `packages/chat/node-crypto-shim.js`.
`default` maps to the pure-JS build so the package is safe under any
bundler that sets none of the three conditions.
The API is synchronous because the sole in-graph consumer,
`makeBlobRefExo`, content-addresses inside a synchronous exo factory;
WebCrypto's async `crypto.subtle.digest` cannot back it. Streaming stays
out of scope: it is already served by the injected
`CryptoPowers.makeSha256`.
Tests cross-check all three builds against `node:crypto` over FIPS 180-4
vectors and lengths straddling the block and padding boundaries, and
exercise the xs build against a stand-in host implementing the Rust
contract.
Design: designs/platform-neutral-hash.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`blob-ref.js` is on the XS daemon bundle's compartment graph, and its static `node:crypto` import was the named blocker for generating `daemon_bootstrap.js`. Swapping it for `@endo/sha256` is behavior-preserving: `encodeBase64` already operated over the digest as a `Uint8Array` rather than through `Buffer.prototype.toString`, so the recorded hash is byte-for-byte what it was. Design: designs/platform-neutral-hash.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shim carried its own 100-line pure-JS SHA-256, which is now `@endo/sha256`'s browser build. The shim keeps only the `createHash` streaming shape that `node:crypto` callers expect and takes the digest from the package, so the two cannot drift apart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Marks the design In Progress, resolves its three open questions as the implementation resolved them, and records the two blockers the original survey did not anticipate: the `@endo/platform/fs/extended` index leg reached through `@endo/exo-git`, and the `@endo/git` / `@endo/host-spawner` exclusion, which turned out to be a prerequisite for generating the bundle rather than a parallel workstream. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Correctness, from the corner-prober seat: - The preferred one-shot XS host path returned whatever the host handed back. `new Uint8Array` coerces rather than refuses, so a host returning the number 32 produced thirty-two zero bytes: one content address shared by every blob, silently. Every digest arriving from a host function is now type- and length-checked, and `sha256Into` checks before writing so a short digest cannot leave stale destination bytes behind a return value of 32. - The streaming triple leaked a host handle when `hostSha256Update Bytes` threw, since `hostSha256Finish` is what consumes it. - The pure-JS padded length used `>> 6`, which coerces to signed 32-bit and goes negative at 2**31 - 8 bytes, throwing where the node build hashes happily. Computed by division now: the builds must not diverge on any input. Tests: fast-check properties over generated input for the cross-build-agreement claim the header states universally and the suite checked at eight lengths, plus host stand-ins returning a short buffer, a long buffer, a number, and a string, plus a throwing update that must not leak a handle. The generated cases replace unseeded `randomBytes`, so a CI failure now reports a seed and a shrunk input. Changesets for the three published surfaces (the new package, the `@endo/platform` shim requirement dropped, the `@endo/daemon` powers field), which the changeset-auditor seat blocked on. Documentation: - `rust/endo/README.md` still described this bundle as failing and prescribed the fix as future work, with specifics the build falsified, and `host-tool-powers.js` cited it as the durable record. - The design's "Open questions" held three answered questions with their deliberation intact; they are resolved questions now, and the deferred host function's trigger names an owner. - Dropped the `sha256Into` allocation claim, which was backwards: it saves a 32-byte array and still allocates its padded input. - Trimmed the WebCrypto rationale from five statements to one, and the README API block from pasted JSDoc to prose signatures. - Said why the sibling XS bundle generators do not pass the `xs` condition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three seats independently ran the same class of attack and it worked: a value that answers a different length to each read is not caught by `instanceof`, and the pure-JS build read `bytes.length` four separate times. A `Proxy` over a real `Uint8Array` returned the empty-input digest for eight bytes of content while the node build threw on the same value, so the package's central claim (the builds never diverge) was false, and in a package whose only in-graph consumer is content addressing a wrong digest is a wrong address, not an error. - `assertBytes` now brand-checks through `%TypedArray%.prototype`'s `length` accessor, captured at module load, which refuses a proxy, ignores a subclass's own `length`, and still accepts a `Buffer`. `jsSha256Into` reads that length once and uses the snapshot. - A detached buffer digested as empty on node and threw V8's own error on the pure-JS build. It now digests as empty everywhere, which is what `node:crypto` does, pinned as deliberate. - The `hostSha256Bytes` digest was aliased, not copied, so a host reusing a scratch buffer could mutate an already-published content address. `.slice()`, as `sha256-node.js` already did off its `Buffer`. - The no-host fallback no longer memoizes, so a digest taken before host-power registration cannot pin pure JS for the process. - The 64-bit bit length is written with `setBigUint64` rather than decomposed into two 32-bit halves. Coverage the prover seat broke and found nothing reddening: `makeSha256Into`'s digest guard, every conditional-export arm (it redirected `xs` at a typo and all 63 tests still passed), and `provideHostToolPowers`' refusal diagnostics, which no daemon test could reach because every Node supervisor supplies all three tools. Each now has a test. CI also generates the daemon bundle now: it is gitignored and was ungated, so a re-introduced `node:` import in the daemon graph reddened nothing. `HostToolPowers` is typed from the implementations rather than `(...args: any[]) => any`, which the locksmith seat flagged as erasing the contract at exactly the seam this refactor introduces. Type-only imports, so no `node:` builtin reaches the XS graph. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The TypeDoc CI gate type-checks JavaScript that `packages/daemon`'s own tsconfig does not (`checkJs: false`), so this file passed the package's `lint:types` and reddened `test`. Now that `HostToolPowers` is typed from the real implementations, calling a tool with no arguments is a type error; the test calls them through a loosely typed view, since what it pins is that a call is refused before any argument is read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
|
Implemented at
Verification: model |
Draft
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs: #124
Description
Restore all three XS bootstrap bundles on the current
llmbase. The daemon core now receives Node-only git and host-spawning implementations as injected powers, while XS receives explicit refusing stand-ins. Platform blob hashing moves behind a new platform-neutral@endo/sha256package so the XS condition selects the Rust host digest without retainingnode:crypto. The rootyarn bundle:xscommand builds SES boot, worker bootstrap, and daemon bootstrap together, and the existing xsnap CI job runs that command as a required step.Security Considerations
The XS daemon gains no host-process authority. Git and shell formulas explicitly refuse when the XS supervisor does not supply host-tool powers. Node supervisors retain their existing implementations. The SHA-256 abstraction validates byte inputs and digest lengths at the boundary.
Scaling Considerations
There is no expected material scaling change. XS uses the existing native host SHA-256 operation; browser fallback uses a bounded synchronous implementation.
Documentation Considerations
The platform-neutral hashing design and Rust/XS bootstrap notes now describe conditional resolution and unavailable host tools. No operator migration is required.
Testing Considerations
yarn bundle:xsbuilds all three artifacts. New SHA-256 unit, property, conditional-export, and boundary tests pass; host-tool refusal tests cover the XS seam. The existing xsnap CI job now runs the daemon bundle too, so a future Node-only import regression fails CI.Compatibility Considerations
Node daemon behavior is unchanged. The new
@endo/sha256package is additive, and platform BlobRef hashes remain byte-for-byte SHA-256 base64 values.Upgrade Considerations
No stored data or wire format changes. Existing content addresses remain compatible.