fix(ci): time-budget graceful stop for sanitized fuzz + rebase budgets on worst measured throughput - #153
Merged
Merged
Conversation
…s on worst measured throughput Daily sanitized-fuzz tripped the ASan inner 90m timeout on 8 of the 10 runs between 2026-06-24 and 07-04 (#142 #144 #146-152), each time with 5-11 % of rounds left (2658-2859/3000 at ~0.5 rnd/s). The pass was killed mid-run, produced no `Results:` summary, and the day's entire ASan coverage signal was lost — 2800 green rounds count for nothing under the all-or-nothing scheme. TSan was next in line: 07-03 finished 63m52s against its 65m limit. Two-layer fix: 1. differential-fuzz.py grows --time-budget-seconds. When the budget is exhausted the loop stops launching new rounds and emits the normal Results: summary, annotated "N/M rounds (time budget)" so partial coverage is explicit and valid. Slow runners now degrade gracefully instead of losing the pass. Default 0 keeps the flag opt-in; ci.yml / nightly-diff-fuzz.yml behavior is unchanged. 2. daily-sanitized-fuzz.yml re-derives budgets from the worst measured throughput (2.03 s/round ASan, 1.92 s/round TSan — the old 1.5/1.6 s figures were fast-day numbers): ASan: budget 105m, outer timeout 90m→110m, step 95m→115m TSan: budget 75m, outer timeout 65m→80m, step 70m→85m job: 180m→215m The in-script budget is the pacing mechanism; the outer timeout demotes to pure hang protection (a wedged engine subprocess or a stalled script), so "incomplete" in the evaluate step now carries a single unambiguous meaning. Evaluate also parses actual rounds from the Results: line so the summary table shows N/M on budget-stopped days, and the incomplete-pass issue template's triage guidance is rewritten for the new semantics. Closes #142 #144 #146 #147 #148 #149 #150 #151 #152.
- guides/ci-quality-baseline.md: add daily-sanitized-fuzz workflow entry (ASan/TSan deep-diagnostic vs nightly Release throughput split), the --time-budget-seconds semantics + "grep Results: consumers before changing output format" warning, and a dedicated section on the two-layer timeout design (in-script pacing / outer hang-protection) with the single-meaning incomplete signal. - index.md: sync ci-quality-baseline entry; add reflection entry. - memory/reflections/sanitized-fuzz-time-budget-graceful-stop.md: new reflection — all-or-nothing structural flaw, worst-case-not-fast-day budget calibration (55 % fast/slow runner variance), two-layer timeout separation, observation-window decision trees need an owner.
Contributor
🔍 PR 审查
两层 timeout(内层
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Daily sanitized-fuzz 的 ASan pass 在 2026-06-24 至 07-04 的 10 次 run 中 8 次被内层 90m timeout 杀死(exit=124),每次只差 5-11%(2658-2859/3000 轮);被杀时无
Results:汇总,整个 pass 的覆盖信号全损,机器人连开 9 个同模板 issue。TSan 是下一个要炸的:07-03 距 65m 上限不足 1 分钟。根因两层:
timeout NNm python3 …超时即 SIGTERM,2800+ 轮全 pass 也一并作废——任何静态数字都会被 runner 方差击穿。Changes
scripts/differential-fuzz.py:新增--time-budget-seconds(默认 0=关闭,ci.yml / nightly-diff-fuzz.yml 行为不变)。预算耗尽即停止发起新轮,正常输出Results:汇总并标注N/M rounds (time budget)。^Results:前缀保持稳定(nightly/daily 两处 evaluate step 都 key 在该前缀上)。.github/workflows/daily-sanitized-fuzz.yml:timeout降级为纯 hang 保护——incomplete 从此只剩一种含义(真 wedge 或脚本 crash)。Results:行解析实际轮数(N/M)进 summary 表;incomplete issue 模板 triage 指引按新语义重写。README.md:daily sanitized fuzz 一行补充 time-budget 语义。llmdoc/:ci-quality-baseline 指南补 daily-sanitized-fuzz 条目与两层 timeout 设计说明;新增 reflection。Test plan
ruff check通过;YAML 结构校验通过(job/step timeout 层级正确)Results: 20 rounds, seed=42,exit=0Results: 56/500 rounds (time budget), seed=42,exit=0--stability-runs 3(CI 形态)+ 预算:正常收尾,UNSTABLE 行照常输出Issues
Closes #142. (#144 / #146-152 已作为 duplicate 关闭指向 #142,trend 分析见 #142 评论。)