fix(cli): atomically publish fat archive cache - #5700
Conversation
# 影响范围 仅修改 CLI fat link 的 `libdeps-<hash>.a` 缓存校验与生成路径;入口为 `BuildRequest::run_fat_link`,未触及 core、renderer、协议或用户项目产物布局。 # 改动影响面 缓存命中现在解析 ar 结构并确认至少一个条目与完整文件边界;无效、空或截断文件会重新生成。生成阶段使用目标目录内唯一临时文件,Darwin ranlib 在临时文件上完成,sync 后通过 tempfile persist 原子替换;并发发布或 Windows 目标占用失败时,仅复用已验证的完整缓存。 # 功能改进/开发/新增 避免中断写入永久污染 fat-binary 缓存;普通失败会自动清理临时文件,SIGKILL 最多留下不参与命中的隐藏唯一临时文件。校验按 archive 条目 seek 扫描,无需把缓存整体载入内存。新增不存在、空、截断和有效 ar 四类单元测试。 # 验证 - `cargo fmt --check`:通过 - `cargo test -p dioxus-cli build::link::tests`:4 passed,15.76s(最终聚焦运行) - `cargo test -p dioxus-cli`:163 passed,16.43s - `cargo clippy -p dioxus-cli --tests -- -D warnings`:通过,10.76s(最终运行) - `git diff --check`:通过
# 影响范围 仅补强 CLI fat archive 缓存校验及其单元测试,不改变缓存路径或 linker 参数。 # 改动影响面 校验在解析前拒绝小于 ar 全局头或不满足两字节成员对齐的文件,因此最后一个 padding byte 被中断时也不会命中缓存。 # 功能改进/开发/新增 新增奇数字节对象成员缺失最终 padding 的回归用例,覆盖此前结构解析器会容忍的 EOF 边界。 # 验证 - `cargo fmt --check`:通过 - `cargo test -p dioxus-cli build::link::tests`:5 passed,20.84s - `cargo clippy -p dioxus-cli --tests -- -D warnings`:通过,16.55s - `git diff --check`:通过
|
I found several cache-transaction and validation edge cases during a deeper review, so I am moving this back to draft while I strengthen the implementation. I am working on transactional archive/sidecar publication, bounded ar parsing, explicit concurrency ownership, Darwin ranlib durability, stale staging cleanup, and failure-injection/concurrency tests. I will keep the PR draft until those invariants are covered and revalidated. |
# 影响范围 仅修改 `BuildRequest::run_fat_link` 的 `libdeps-<hash>.a`、`rlibs-<hash>.txt` 缓存协议及 `link.rs` 内部校验/事务 helper;linker 参数、最终可执行文件、Dioxus core 与 renderer 不受影响。 # 改动影响面 同 hash 构建通过 OS 文件锁串行化。archive、sidecar 与 manifest 均在目标目录 staging,按 sidecar -> archive -> manifest commit marker 顺序原子发布;命中必须同时匹配 archive 结构指纹/长度与 sidecar SHA256/长度。重建先使 marker 失效,debug 强制重建不复用旧缓存,persist/权限错误不再按目标文件存在而吞掉。锁内只清理当前 hash 的遗留 staging,避免误删活跃并发文件。 # 功能改进/开发/新增 在交给 ar crate 前执行有界结构扫描,对全局头、每个 header、成员/BSD 名称/GNU 名称偏移与 padding 使用 checked_add 和 file_len 边界;语义校验要求至少一个非零 `.rcgu.o`/`.obj` 且拒绝其他普通成员与特殊表-only archive。Darwin ranlib 后按路径重新打开并 sync,manifest 发布后同步父目录。新增事务失败注入、并发锁、persist 错误、stale temp、GNU/BSD/截断/OOM 边界及真实 macOS ranlib 测试。 # 验证 - `cargo test -p dioxus-cli build::link::tests`:21 passed,7.21s - `cargo test -p dioxus-cli`:180 passed,13.30s - `cargo clippy -p dioxus-cli --tests -- -D warnings`:通过,10.38s - `cargo fmt --check`:通过 - `git diff --check`:通过
|
The transaction/validation hardening is now pushed in The focused suite now has 21 tests covering transaction failure injection, concurrency, persist failure, stale staging, bounded GNU/BSD parsing, special-table-only archives, zero/wrong members, and real macOS ranlib behavior. Full CLI tests (180) and clippy with |
|
A second review found two remaining durability/lifecycle gaps, so I am moving the PR back to draft again. I am tightening the publish ordering and archive identity, and extending the per-hash lock through linker completion so a failed fat link cannot leave a committed cache transaction or race with a debug rebuild. I will also narrow the documented guarantee to atomic visibility/process interruption rather than claiming cross-platform power-loss durability. |
# 影响范围 仅修改 dioxus-cli fat archive 缓存的校验、链接与发布生命周期,入口为 BuildRequest 的 fat linking 路径。 # 改动影响面 同一 hash 的 OS 文件锁现在覆盖缓存校验、staged archive 链接与事务提交;缓存命中增加完整 archive SHA-256 校验,时间复杂度为 O(file bytes)、空间复杂度为 O(1) 流式缓冲。domain、renderer、协议与普通 thin link 路径不受影响。 # 功能改进/开发/新增 先链接 staged archive,只有 linker 成功后才按 sidecar、archive、目录同步、manifest marker、目录同步的顺序提交。linker 选择、命令文件、进程启动或退出失败时在锁内移除 marker。manifest v2 绑定完整 archive 内容哈希;Unix 提供目录 fsync 的掉电顺序约束,Windows 仅承诺原子可见性与进程中断恢复,不宣称通用掉电持久性。新增 payload 同步前禁止发布 marker、完整内容替换失效与 linker 失败 marker 失效测试。 # 验证 cargo test -p dioxus-cli build::link::tests:23 passed,15.32s wall-clock。 cargo test -p dioxus-cli:182 passed,14.13s wall-clock。 cargo clippy -p dioxus-cli --tests -- -D warnings:通过,19.14s wall-clock。 cargo fmt --check:通过。 git diff --check:通过。
|
The second-review lifecycle and durability follow-up is now pushed in New archives are linked from their staged path and are committed only after the fat linker succeeds. The per-hash OS lock now remains held through linking and commit; linker selection, response-file creation, process execution, and nonzero exit failures invalidate the manifest marker under that lock. Publication is ordered as payload renames, Unix parent-directory sync, manifest rename, then a second directory sync. The v2 manifest now binds a streaming SHA-256 of the complete archive, so cache-hit validation deliberately costs O(archive bytes) I/O with O(1) memory. Documentation now limits the Windows guarantee to atomic visibility/process-interruption recovery and does not claim general power-loss durability. Focused link tests pass 23/23, the full dioxus-cli suite passes 182/182, clippy passes with |
Summary
libdeps-<hash>.a,rlibs-<hash>.txt, and a versioned manifest as one cache transaction protected by a per-hash OS file lockarcrate, then require every ordinary member to be a non-empty.rcgu.oor.objLink and transaction lifecycle
All readers and writers for the same hash acquire
.fat-cache-<hash>.lock, and the lock remains held through cache validation, fat linking, and any subsequent commit. Rebuilds link the staged archive directly. Only a successful linker run may commit it; linker selection, response-file creation, process execution, or nonzero exit failures remove and sync the manifest marker while still holding the lock.This also keeps debug rebuilds serialized and prevents them from replacing an archive another process is currently linking. Startup removes only the locked hash's uniquely named staging files, which cleans up interrupted writes without deleting an active writer's files.
Publication and durability
The manifest is the commit marker. Publication order is:
Readers accept the transaction only when the manifest's lengths and hashes match both payloads. The v2 manifest binds the complete archive SHA-256 in addition to its structural fingerprint. This deliberately makes cache-hit validation O(archive bytes) I/O, with O(1) streaming memory, in exchange for detecting same-layout content replacement.
On Unix, the directory syncs preserve the intended payload-before-marker ordering across process interruption and power loss, subject to the filesystem's guarantees. On Windows, publication still provides atomic visibility/process-interruption recovery through same-directory replacement and the last-published marker, but this PR does not claim general power-loss durability because parent-directory fsync is unavailable here.
Bounded archive validation
Before
ar::Archivesees the file, the scanner validates the global header, every 60-byte member header, decimal size, BSD extended-name size, GNU name-table reference, member body, and padding. All offsets use checked arithmetic and must remain within the actual file length, preventing a tiny forged archive from triggering a huge allocation in the parser.The semantic pass rejects zero-length objects, wrong ordinary member types, and GNU/BSD symbol- or name-table-only archives. It accepts valid common, BSD long-name, and GNU long-name archives.
Darwin ranlib
Darwin
ranlibruns on the staged archive before linking. Because it may replace the inode, dx reopens the staged path and syncs that inode instead of relying on the pre-ranlibNamedTempFilehandle. A macOS test builds a real object archive, runs the systemranlib, reopens/syncs it, and validates the result.Tests
cargo test -p dioxus-cli build::link::tests(23 passed)cargo test -p dioxus-cli(182 passed)cargo clippy -p dioxus-cli --tests -- -D warningscargo fmt --checkgit diff --checkFixes #5684