Skip to content

fix(ci): time-budget graceful stop for sanitized fuzz + rebase budgets on worst measured throughput - #153

Merged
Liam0205 merged 2 commits into
masterfrom
fix/sanitized-fuzz-time-budget
Jul 5, 2026
Merged

fix(ci): time-budget graceful stop for sanitized fuzz + rebase budgets on worst measured throughput#153
Liam0205 merged 2 commits into
masterfrom
fix/sanitized-fuzz-time-budget

Conversation

@Liam0205

@Liam0205 Liam0205 commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Summary

Daily sanitized-fuzz 的 ASan pass 在 2026-06-24 至 07-04 的 10 次 run 中 8 次被内层 90m timeout 杀死(exit=124),每次只差 5-11%(2658-2859/3000 轮);被杀时无 Results: 汇总,整个 pass 的覆盖信号全损,机器人连开 9 个同模板 issue。TSan 是下一个要炸的:07-03 距 65m 上限不足 1 分钟。

根因两层:

  1. 预算基于快日标定:注释里的 ~1.5/1.6 s/round 来自 Daily sanitized-fuzz: pass incomplete (ASan=incomplete TSan=incomplete) (2026-06-20) #132/Daily sanitized-fuzz: pass incomplete (ASan=incomplete TSan=incomplete) (2026-06-21) #133 两天窗口;10 天窗口的 worst 实测是 ~2.03/1.92 s/round(快慢日方差 ~55%)。
  2. all-or-nothing 结构timeout NNm python3 … 超时即 SIGTERM,2800+ 轮全 pass 也一并作废——任何静态数字都会被 runner 方差击穿。

Changes

  • scripts/differential-fuzz.py:新增 --time-budget-seconds(默认 0=关闭,ci.yml / nightly-diff-fuzz.yml 行为不变)。预算耗尽即停止发起新轮,正常输出 Results: 汇总并标注 N/M rounds (time budget)^Results: 前缀保持稳定(nightly/daily 两处 evaluate step 都 key 在该前缀上)。
  • .github/workflows/daily-sanitized-fuzz.yml
    • 按 worst 实测重标:ASan in-script budget 105m / outer timeout 90→110m / step 95→115m;TSan 75m / 65→80m / 70→85m;job 180→215m。
    • 语义变化:in-script budget 承担 pacing,外层 timeout 降级为纯 hang 保护——incomplete 从此只剩一种含义(真 wedge 或脚本 crash)。
    • evaluate step 从 Results: 行解析实际轮数(N/M)进 summary 表;incomplete issue 模板 triage 指引按新语义重写。
  • README.md:daily sanitized fuzz 一行补充 time-budget 语义。
  • llmdoc/:ci-quality-baseline 指南补 daily-sanitized-fuzz 条目与两层 timeout 设计说明;新增 reflection。

Test plan

  • ruff check 通过;YAML 结构校验通过(job/step timeout 层级正确)
  • 无预算 20 轮(go+java):行为与旧版一致,Results: 20 rounds, seed=42,exit=0
  • 500 轮 + 15s 预算:56 轮后优雅停止,Results: 56/500 rounds (time budget), seed=42,exit=0
  • --stability-runs 3(CI 形态)+ 预算:正常收尾,UNSTABLE 行照常输出
  • 用真实日志端到端模拟 evaluate step 的 4 个 grep 模式(FAIL / UNSTABLE / HAS_RESULTS / parse_rounds,含缺失文件 fallback)

Issues

Closes #142. (#144 / #146-152 已作为 duplicate 关闭指向 #142,trend 分析见 #142 评论。)

Liam0205 added 2 commits July 5, 2026 15:09
…s on worst measured throughput

Daily sanitized-fuzz tripped the ASan inner 90m timeout on 8 of the 10
runs between 2026-06-24 and 07-04 (#142 #144 #146-152), each time with
5-11 % of rounds left (2658-2859/3000 at ~0.5 rnd/s). The pass was
killed mid-run, produced no `Results:` summary, and the day's entire
ASan coverage signal was lost — 2800 green rounds count for nothing
under the all-or-nothing scheme. TSan was next in line: 07-03 finished
63m52s against its 65m limit.

Two-layer fix:

1. differential-fuzz.py grows --time-budget-seconds. When the budget
   is exhausted the loop stops launching new rounds and emits the
   normal Results: summary, annotated "N/M rounds (time budget)" so
   partial coverage is explicit and valid. Slow runners now degrade
   gracefully instead of losing the pass. Default 0 keeps the flag
   opt-in; ci.yml / nightly-diff-fuzz.yml behavior is unchanged.

2. daily-sanitized-fuzz.yml re-derives budgets from the worst
   measured throughput (2.03 s/round ASan, 1.92 s/round TSan — the
   old 1.5/1.6 s figures were fast-day numbers):

     ASan: budget 105m, outer timeout 90m→110m, step 95m→115m
     TSan: budget 75m,  outer timeout 65m→80m,  step 70m→85m
     job: 180m→215m

   The in-script budget is the pacing mechanism; the outer timeout
   demotes to pure hang protection (a wedged engine subprocess or a
   stalled script), so "incomplete" in the evaluate step now carries
   a single unambiguous meaning. Evaluate also parses actual rounds
   from the Results: line so the summary table shows N/M on
   budget-stopped days, and the incomplete-pass issue template's
   triage guidance is rewritten for the new semantics.

Closes #142 #144 #146 #147 #148 #149 #150 #151 #152.
- guides/ci-quality-baseline.md: add daily-sanitized-fuzz workflow entry
  (ASan/TSan deep-diagnostic vs nightly Release throughput split), the
  --time-budget-seconds semantics + "grep Results: consumers before
  changing output format" warning, and a dedicated section on the
  two-layer timeout design (in-script pacing / outer hang-protection)
  with the single-meaning incomplete signal.
- index.md: sync ci-quality-baseline entry; add reflection entry.
- memory/reflections/sanitized-fuzz-time-budget-graceful-stop.md: new
  reflection — all-or-nothing structural flaw, worst-case-not-fast-day
  budget calibration (55 % fast/slow runner variance), two-layer
  timeout separation, observation-window decision trees need an owner.
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

🔍 PR 审查

项目 结果
结论 ✅ APPROVE
审查截止 14f31d034c881d4ce4526602996805dd196b0d52

两层 timeout(内层 --time-budget-seconds pacing + 外层纯 hang 保护)设计正确,实现与文档一致,未发现 bug 或规范问题。

✅ 验证要点

逐项核对,均通过:

  • start_time 生命周期:定义于 differential-fuzz.py:1386,早于循环内 budget 检查 L1409-L1421,无未定义引用。
  • 失败信号保留:budget 触发只是 break,随后照常走 sys.exit(1 if failed > 0 else 0)——提前收束不会吞掉已发现的 divergence。且 daily workflow 用 grep FAIL: 计数判定失败(非 exit code),双保险。
  • ^Results: 前缀契约稳定:改动只在前缀之后扩展 N/M rounds (time budget)L1671-L1674),两处消费者(nightly-diff-fuzz.ymlgrep -c '^Results:'daily-sanitized-fuzz.yml L212-213)不受影响。
  • evaluate 解析健壮parse_rounds[0-9/]+ 同时匹配 56/50020 两种形态,无 Results 行时 fallback 到 planned 值。
  • budget/timeout 分层自洽:ASan 6300s(105m) < outer 110m < step 115m;TSan 4500s(75m) < outer 80m < step 85m;job 215m 覆盖 7+115+85+5≈212m。数值与头部注释、commit message、README、llmdoc 全部一致。
  • 默认 opt-in 无回归--time-budget-seconds 默认 0(关闭),ci.yml / nightly-diff-fuzz.yml 未传该参数,行为不变。
  • 提前停止提示行 以两空格缩进开头,不会误撞 ^Results: / ^FAIL: 等消费 grep。

Submodule 无变更;llmdoc 同步(ci-quality-baseline 指南、index、新增 reflection)完整且未硬编码易漂移的分钟数。

@Liam0205
Liam0205 merged commit d6a7e32 into master Jul 5, 2026
21 checks passed
@Liam0205
Liam0205 deleted the fix/sanitized-fuzz-time-budget branch July 5, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daily sanitized-fuzz: pass incomplete (ASan=incomplete TSan=complete) (2026-06-24)

1 participant