Per-package atomic publish: metas travel with bytes, ref registers last#52
Merged
Conversation
A publish run cancelled mid-way (re-runs auto-cancel in-flight attempts) left R2 tarballs overwritten with new bytes while the packument kept serving the previous metas for ~20 minutes, failing every install in the window with ERR_PNPM_TARBALL_INTEGRITY and poisoning CDN edge caches and pnpm client metadata caches with the mixed state (2026-07-02 incident, sha 0fecc75a). The action now publishes each package's meta (a new register:false mode of /-/publish) immediately after uploading its tarball, so bytes and meta can never diverge for longer than one package's upload, and registers the ref in one final packages-empty call so a new version flips visible atomically. The old single-call protocol still works unchanged. Also fix the on-demand rebuild writing only the per-version meta: it now updates the meta-index too, so the packument cannot flap between two meta sources that disagree (an in-worker workerd gzip never matches CI's Node gzip byte-for-byte, so a divergent stale index entry advertised an integrity the served bytes could no longer satisfy).
Without a deadline, a hung connection stalls up to undici's ~5-minute defaults, and 4 retry attempts across ~11 packages can zombie a publish for tens of minutes (the incident's cancelled attempt ran 21 minutes mid-publish, which is what made mid-publish cancellation likely in the first place). Transfers (pkg.pr.new download, tarball PUT, up to ~19MB) get 120s per attempt; the small register/publish POST gets 30s.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident (2026-07-02, sha 0fecc75a)
A publish run was cancelled mid-way (re-running a workflow auto-cancels the in-flight attempt). It had already overwritten the R2 tarballs for vite-plus/core with new bytes, but never reached the single end-of-run
/-/publish, so for ~20 minutes the packument advertised metas that no longer matched the served bytes. Every install in the window failed withERR_PNPM_TARBALL_INTEGRITY, and the mixed state got captured by CDN edge caches (immutable, deployment-scoped) and pnpm client metadata caches, both of which outlived the server-side self-heal.Fix
/-/publishacceptsregister: false(store metas only, ref stays unregistered/invisible) and a finalpackages: []register-only call. The old single-call protocol is unchanged.buildAndStore) now updates the meta-index alongside the per-version meta. Previously it skipped the index, leaving two disagreeing meta sources: the packument flapped between them (workerd's gzip output never matches CI's Node gzip, so a stale index integrity was unsatisfiable by the rebuilt bytes).Test plan
pnpm test81/81,pnpm typecheckpnpm build:action)Follow-ups (separate)
dist.tarballso a poisoned edge/client cache can never outlive a fix (CDN ignores query strings, so it must be path-based)continue-on-error: trueon the register step and cancellation of in-flight publishes