Skip to content

fix(cli): atomically publish fat archive cache - #5700

Open
zxs1633079383 wants to merge 4 commits into
DioxusLabs:mainfrom
zxs1633079383:codex/issue-5684-atomic-fat-cache
Open

fix(cli): atomically publish fat archive cache#5700
zxs1633079383 wants to merge 4 commits into
DioxusLabs:mainfrom
zxs1633079383:codex/issue-5684-atomic-fat-cache

Conversation

@zxs1633079383

@zxs1633079383 zxs1633079383 commented Jul 23, 2026

Copy link
Copy Markdown

Summary

  • treat libdeps-<hash>.a, rlibs-<hash>.txt, and a versioned manifest as one cache transaction protected by a per-hash OS file lock
  • link a newly staged archive before publishing it, and publish the transaction only after the fat linker succeeds
  • reject missing, incomplete, mismatched, empty, malformed, truncated, or content-modified cache transactions and rebuild them automatically
  • perform a bounded ar structural scan before invoking the ar crate, then require every ordinary member to be a non-empty .rcgu.o or .obj

Link and transaction lifecycle

All readers and writers for the same hash acquire .fat-cache-<hash>.lock, and the lock remains held through cache validation, fat linking, and any subsequent commit. Rebuilds link the staged archive directly. Only a successful linker run may commit it; linker selection, response-file creation, process execution, or nonzero exit failures remove and sync the manifest marker while still holding the lock.

This also keeps debug rebuilds serialized and prevents them from replacing an archive another process is currently linking. Startup removes only the locked hash's uniquely named staging files, which cleans up interrupted writes without deleting an active writer's files.

Publication and durability

The manifest is the commit marker. Publication order is:

  1. rename the sidecar
  2. rename the archive
  3. sync the parent directory on Unix
  4. rename the manifest marker
  5. sync the parent directory again on Unix

Readers accept the transaction only when the manifest's lengths and hashes match both payloads. The v2 manifest binds the complete archive SHA-256 in addition to its structural fingerprint. This deliberately makes cache-hit validation O(archive bytes) I/O, with O(1) streaming memory, in exchange for detecting same-layout content replacement.

On Unix, the directory syncs preserve the intended payload-before-marker ordering across process interruption and power loss, subject to the filesystem's guarantees. On Windows, publication still provides atomic visibility/process-interruption recovery through same-directory replacement and the last-published marker, but this PR does not claim general power-loss durability because parent-directory fsync is unavailable here.

Bounded archive validation

Before ar::Archive sees the file, the scanner validates the global header, every 60-byte member header, decimal size, BSD extended-name size, GNU name-table reference, member body, and padding. All offsets use checked arithmetic and must remain within the actual file length, preventing a tiny forged archive from triggering a huge allocation in the parser.

The semantic pass rejects zero-length objects, wrong ordinary member types, and GNU/BSD symbol- or name-table-only archives. It accepts valid common, BSD long-name, and GNU long-name archives.

Darwin ranlib

Darwin ranlib runs on the staged archive before linking. Because it may replace the inode, dx reopens the staged path and syncs that inode instead of relying on the pre-ranlib NamedTempFile handle. A macOS test builds a real object archive, runs the system ranlib, reopens/syncs it, and validates the result.

Tests

  • cargo test -p dioxus-cli build::link::tests (23 passed)
  • cargo test -p dioxus-cli (182 passed)
  • cargo clippy -p dioxus-cli --tests -- -D warnings
  • cargo fmt --check
  • git diff --check

Fixes #5684

# 影响范围
仅修改 CLI fat link 的 `libdeps-<hash>.a` 缓存校验与生成路径;入口为 `BuildRequest::run_fat_link`,未触及 core、renderer、协议或用户项目产物布局。

# 改动影响面
缓存命中现在解析 ar 结构并确认至少一个条目与完整文件边界;无效、空或截断文件会重新生成。生成阶段使用目标目录内唯一临时文件,Darwin ranlib 在临时文件上完成,sync 后通过 tempfile persist 原子替换;并发发布或 Windows 目标占用失败时,仅复用已验证的完整缓存。

# 功能改进/开发/新增
避免中断写入永久污染 fat-binary 缓存;普通失败会自动清理临时文件,SIGKILL 最多留下不参与命中的隐藏唯一临时文件。校验按 archive 条目 seek 扫描,无需把缓存整体载入内存。新增不存在、空、截断和有效 ar 四类单元测试。

# 验证
- `cargo fmt --check`:通过
- `cargo test -p dioxus-cli build::link::tests`:4 passed,15.76s(最终聚焦运行)
- `cargo test -p dioxus-cli`:163 passed,16.43s
- `cargo clippy -p dioxus-cli --tests -- -D warnings`:通过,10.76s(最终运行)
- `git diff --check`:通过
@zxs1633079383
zxs1633079383 marked this pull request as ready for review July 23, 2026 07:38
@zxs1633079383
zxs1633079383 requested a review from a team as a code owner July 23, 2026 07:38
# 影响范围
仅补强 CLI fat archive 缓存校验及其单元测试,不改变缓存路径或 linker 参数。

# 改动影响面
校验在解析前拒绝小于 ar 全局头或不满足两字节成员对齐的文件,因此最后一个 padding byte 被中断时也不会命中缓存。

# 功能改进/开发/新增
新增奇数字节对象成员缺失最终 padding 的回归用例,覆盖此前结构解析器会容忍的 EOF 边界。

# 验证
- `cargo fmt --check`:通过
- `cargo test -p dioxus-cli build::link::tests`:5 passed,20.84s
- `cargo clippy -p dioxus-cli --tests -- -D warnings`:通过,16.55s
- `git diff --check`:通过
@zxs1633079383 zxs1633079383 reopened this Jul 23, 2026
@zxs1633079383
zxs1633079383 marked this pull request as draft July 23, 2026 07:52
@zxs1633079383

Copy link
Copy Markdown
Author

I found several cache-transaction and validation edge cases during a deeper review, so I am moving this back to draft while I strengthen the implementation. I am working on transactional archive/sidecar publication, bounded ar parsing, explicit concurrency ownership, Darwin ranlib durability, stale staging cleanup, and failure-injection/concurrency tests. I will keep the PR draft until those invariants are covered and revalidated.

# 影响范围
仅修改 `BuildRequest::run_fat_link` 的 `libdeps-<hash>.a`、`rlibs-<hash>.txt` 缓存协议及 `link.rs` 内部校验/事务 helper;linker 参数、最终可执行文件、Dioxus core 与 renderer 不受影响。

# 改动影响面
同 hash 构建通过 OS 文件锁串行化。archive、sidecar 与 manifest 均在目标目录 staging,按 sidecar -> archive -> manifest commit marker 顺序原子发布;命中必须同时匹配 archive 结构指纹/长度与 sidecar SHA256/长度。重建先使 marker 失效,debug 强制重建不复用旧缓存,persist/权限错误不再按目标文件存在而吞掉。锁内只清理当前 hash 的遗留 staging,避免误删活跃并发文件。

# 功能改进/开发/新增
在交给 ar crate 前执行有界结构扫描,对全局头、每个 header、成员/BSD 名称/GNU 名称偏移与 padding 使用 checked_add 和 file_len 边界;语义校验要求至少一个非零 `.rcgu.o`/`.obj` 且拒绝其他普通成员与特殊表-only archive。Darwin ranlib 后按路径重新打开并 sync,manifest 发布后同步父目录。新增事务失败注入、并发锁、persist 错误、stale temp、GNU/BSD/截断/OOM 边界及真实 macOS ranlib 测试。

# 验证
- `cargo test -p dioxus-cli build::link::tests`:21 passed,7.21s
- `cargo test -p dioxus-cli`:180 passed,13.30s
- `cargo clippy -p dioxus-cli --tests -- -D warnings`:通过,10.38s
- `cargo fmt --check`:通过
- `git diff --check`:通过
@zxs1633079383

Copy link
Copy Markdown
Author

The transaction/validation hardening is now pushed in 5b08652ae. The cache is serialized by a per-hash OS lock and committed with a manifest marker after the sidecar and archive; partial publication is rejected by matching both fingerprints. Validation now performs checked, file-bounded ar scanning before the parser and enforces non-empty Rust object members. Persist errors propagate, debug rebuilds cannot reuse stale state, same-hash SIGKILL staging is cleaned under the lock, and Darwin ranlib is followed by path reopen/sync plus parent-directory sync.

The focused suite now has 21 tests covering transaction failure injection, concurrency, persist failure, stale staging, bounded GNU/BSD parsing, special-table-only archives, zero/wrong members, and real macOS ranlib behavior. Full CLI tests (180) and clippy with -D warnings pass.

@zxs1633079383

Copy link
Copy Markdown
Author

A second review found two remaining durability/lifecycle gaps, so I am moving the PR back to draft again. I am tightening the publish ordering and archive identity, and extending the per-hash lock through linker completion so a failed fat link cannot leave a committed cache transaction or race with a debug rebuild. I will also narrow the documented guarantee to atomic visibility/process interruption rather than claiming cross-platform power-loss durability.

# 影响范围
仅修改 dioxus-cli fat archive 缓存的校验、链接与发布生命周期,入口为 BuildRequest 的 fat linking 路径。

# 改动影响面
同一 hash 的 OS 文件锁现在覆盖缓存校验、staged archive 链接与事务提交;缓存命中增加完整 archive SHA-256 校验,时间复杂度为 O(file bytes)、空间复杂度为 O(1) 流式缓冲。domain、renderer、协议与普通 thin link 路径不受影响。

# 功能改进/开发/新增
先链接 staged archive,只有 linker 成功后才按 sidecar、archive、目录同步、manifest marker、目录同步的顺序提交。linker 选择、命令文件、进程启动或退出失败时在锁内移除 marker。manifest v2 绑定完整 archive 内容哈希;Unix 提供目录 fsync 的掉电顺序约束,Windows 仅承诺原子可见性与进程中断恢复,不宣称通用掉电持久性。新增 payload 同步前禁止发布 marker、完整内容替换失效与 linker 失败 marker 失效测试。

# 验证
cargo test -p dioxus-cli build::link::tests:23 passed,15.32s wall-clock。
cargo test -p dioxus-cli:182 passed,14.13s wall-clock。
cargo clippy -p dioxus-cli --tests -- -D warnings:通过,19.14s wall-clock。
cargo fmt --check:通过。
git diff --check:通过。
@zxs1633079383

Copy link
Copy Markdown
Author

The second-review lifecycle and durability follow-up is now pushed in efd65f49d.

New archives are linked from their staged path and are committed only after the fat linker succeeds. The per-hash OS lock now remains held through linking and commit; linker selection, response-file creation, process execution, and nonzero exit failures invalidate the manifest marker under that lock. Publication is ordered as payload renames, Unix parent-directory sync, manifest rename, then a second directory sync.

The v2 manifest now binds a streaming SHA-256 of the complete archive, so cache-hit validation deliberately costs O(archive bytes) I/O with O(1) memory. Documentation now limits the Windows guarantee to atomic visibility/process-interruption recovery and does not claim general power-loss durability.

Focused link tests pass 23/23, the full dioxus-cli suite passes 182/182, clippy passes with -D warnings, and fmt/diff checks pass. I am marking the PR ready for review.

@zxs1633079383
zxs1633079383 marked this pull request as ready for review July 23, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cli: interrupted write of libdeps-<hash>.a permanently poisons the fat-binary cache (written in place, cache hit accepts empty archives)

1 participant