-
Notifications
You must be signed in to change notification settings - Fork 6
431 lines (401 loc) · 19.4 KB
/
Copy pathaudit-debt.yml
File metadata and controls
431 lines (401 loc) · 19.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
# Post-merge guard for `main`, plus the weekly full-tree audit debt sweep.
#
# (File name is historical — this started life as the audit-only debt sweep and
# grew the post-merge lint/test gates. Kept as `audit-debt.yml` so the file's
# git history stays intact.)
#
# ── Why the post-merge gates exist (#6807) ──────────────────────────────────
# Two pipelines guard this repo, and until now BOTH were changed-scope:
#
# * ci.yml (PRs) — `differential-gating: true`, scoped to the PR diff.
# * release.yml — gate-lint/gate-test pass `--changed-since <before>`, so
# they only re-check what that one push touched.
#
# Nothing ever ran the whole suite against the integrated `main`. A change whose
# blast radius lands OUTSIDE its own diff is invisible to both: the PR never
# runs the affected tests, and the release only runs them if some later merge
# happens to touch those same files.
#
# That is exactly how 2026-07-27 went. A POSIX-sh portability bug merged green,
# main went red in `crates/homeboy-lab-runner/src/workspace/tests/prune.rs`, and
# nobody found out until merge #10425 happened to touch prune.rs and dragged it
# into the release's changed scope. Release runs 30272803452 and 30271131168
# both died at `Release Quality Policy` → `Test`. Cost: 188 commits
# undeliverable, two orphaned tags, ~16 hours of stranded work.
#
# The release is the WORST place to discover this: it runs 16-35 minutes and a
# failure there can strand a tag. So the gates below run the release-blocking
# command set (RELEASE_BLOCKING_COMMANDS = `review lint,review test`) against
# `main` on every merge, at FULL scope.
#
# Full scope is not an extra flag — homeboy-action's `scope: auto` already
# resolves to `mode=full` on push events. The release narrows itself by
# explicitly passing `--changed-since`; we simply do not pass it. That makes
# this gate a strict superset of the release's blocking scope, which is the
# point: anything that would fail the release is caught here first, and so is
# the cross-scope breakage the release would have missed.
#
# ── Why the weekly sweep exists ─────────────────────────────────────────────
# The PR pipeline runs a `--profile=pr --changed-since` audit: fast, scoped to
# the diff, and it only runs the cheap RootOnly detector families. The
# whole-tree discovery detectors (duplication, dead code, constant/command
# bypass, god files, …) are deliberately OFF in the PR profile — they need the
# entire codebase to work and would re-report repo-wide debt on every unrelated
# PR.
#
# Those discovery detectors are the "code factory" roadmap: each finding kind
# becomes one deduplicated tracking issue (via homeboy-action's auto-issue
# filing), and the issue closes when that fix kind gets automated. The sweep
# runs weekly, decoupled from releases so a full audit never taxes the release
# path, and files/refreshes one issue per finding kind.
name: Main Guard
on:
push:
branches: [main]
schedule:
# Mondays 07:00 UTC — once a week is plenty for slow-moving structural debt.
- cron: '0 7 * * 1'
workflow_dispatch:
inputs:
profile:
description: 'Audit profile to run (full = all discovery detectors).'
required: false
default: 'full'
# Latest merge wins. This gate answers exactly one question — "is main green
# RIGHT NOW?" — and that answer is a pure function of the current tip, so a
# superseded run's verdict is worthless. It also produces no artifacts, mutates
# nothing, and blocks nothing downstream, so cancelling it is free.
#
# This is deliberately the OPPOSITE of release.yml's `cancel-in-progress: false`.
# The release queues because it may be mid-tag/mid-publish and cancelling it
# strands a tag. Nothing here can strand anything.
#
# It is also a capacity requirement, not just a nicety: at ~40 merges/day
# against a 16-35 minute suite, queueing every merge would serialise into
# 10-23 hours of backlog and the gate would end up reporting on hours-old main
# — strictly worse than reporting on the current tip.
#
# Trade-off, stated plainly: when a burst of merges collapses into one run, the
# failure is attributed to the newest merge rather than the one that actually
# broke it. The failure summary prints the compare range so the candidate set is
# still bounded (see "Attribute failure to this merge" below).
#
# Push and sweep get separate groups so a merge train never cancels the weekly
# audit sweep (and vice versa).
concurrency:
group: main-guard-${{ github.event_name == 'push' && 'post-merge' || 'sweep' }}
cancel-in-progress: true
permissions:
contents: read
issues: write
# Shared-artifact handoff between gate-build and the gates.
actions: read
jobs:
# ── Build once, share with all three post-merge gates ──
# Mirrors release.yml's gate-build → artifact → `binary-path` handoff. Before
# this, the audit gate compiled homeboy by itself via `cargo run`; adding two
# more gates that each did their own build would have tripled compile cost on
# every merge. One release build now feeds all three.
gate-build:
name: Build
if: github.event_name == 'push'
runs-on: ubuntu-latest
# Least privilege: this job compiles and uploads, it never files issues.
permissions:
contents: read
actions: read
steps:
- uses: actions/checkout@v6
- uses: dtolnay/rust-toolchain@stable
# Deliberately the same cache key as release.yml's gate-build: both run
# `cargo build --release` on `main`, so whichever lands first warms the
# cache for the other. A key drift in release.yml only costs a cache miss.
- name: Cache cargo
uses: actions/cache@v4
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-release-gate-${{ hashFiles('Cargo.lock') }}
restore-keys: ${{ runner.os }}-cargo-release-gate-
- name: Build homeboy
run: cargo build --release --locked
- name: Upload binary
uses: actions/upload-artifact@v4
with:
name: homeboy-binary
path: target/release/homeboy
retention-days: 1
# ── Full-tree audit gate ──
#
# #10557: this job used to invoke `.homeboy-bin/homeboy review audit` directly.
# That skipped homeboy-action's `Install extension` step, so no extension
# declared `provides.file_extensions`, the audit corpus came back EMPTY, and
# the gate reported `files_scanned: 0, files_skipped: 1817, findings: [],
# passed: true` — green in 7 seconds on an 1800-file repository. The sibling
# lint/test gates below were unaffected because they already go through the
# action, which is why the workflow as a whole looked healthy.
#
# Two changes close it:
#
# 1. Route through Extra-Chill/homeboy-action, exactly like the lint/test
# gates and the weekly sweep, so the extension is installed and the audit
# has something to fingerprint.
# 2. `review audit` itself now hard-errors on an empty corpus
# (engine.rs, "audit.corpus"), so the 7-second green run is impossible
# for EVERY consumer, not just this workflow. The `Assert the audit
# actually scanned` step below is the belt to that braces: it is
# unconditionally blocking and fails closed.
#
# ── Why the findings verdict is reporting-only, for now ──
#
# Turning the findings verdict on today turns `main` red immediately, for debt
# nobody has triaged and that this PR did not create:
#
# * 37 findings on `main` are already unbaselined (`drift_increased: true`,
# 28 resolved, net +9) — accumulated since the 2026-07-27 baseline.
# * ~293 core_boundary_leak findings across 38 files become visible for the
# first time now that the source-policy corpus is fixed (#10558): 182
# index files (`mod.rs`/`lib.rs`/`main.rs`) plus 83 files sitting alone in
# a directory were unscannable by ANY source policy.
#
# Baselining that set requires running the FIXED binary, which only exists
# after this merges. So the findings verdict lands reporting-only and the
# follow-up issue below flips it — with a regenerated baseline — as its own
# reviewed change. `continue-on-error` here is a dated, tracked state, not a
# permanent one; `tests/audit_debt_workflow_test.rs` requires the follow-up
# issue reference to sit next to it so it cannot quietly become permanent.
#
# FOLLOW-UP: #10569 — regenerate the audit baseline on main and remove
# `continue-on-error` from the audit step below.
full-audit-gate:
name: Full-tree audit gate
if: github.event_name == 'push'
needs: gate-build
runs-on: ubuntu-latest
# Pure pass/fail gate — deliberately cannot file issues, unlike the sweep.
permissions:
contents: read
actions: read
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Download homeboy binary
uses: actions/download-artifact@v4
with:
name: homeboy-binary
path: .homeboy-bin
- name: Verify homeboy binary present
run: |
if [ ! -f .homeboy-bin/homeboy ]; then
echo "::error::Build artifact missing from upstream Build job: .homeboy-bin/homeboy was not produced/uploaded by gate-build. This is a CI artifact-handoff problem, not a code finding on main. Re-run the failed Build job or investigate the homeboy-binary upload step." >&2
exit 1
fi
chmod +x .homeboy-bin/homeboy
# Reporting-only until #10569 baselines the pre-existing set. The action
# owns the `Install extension` step whose absence caused the 0-file scan.
- uses: Extra-Chill/homeboy-action@v2
id: audit
continue-on-error: true
with:
binary-path: .homeboy-bin/homeboy
commands: review audit
expected-commands: review audit,review lint,review test
# Full profile enables the whole-tree discovery detectors
# (duplication, dead code, source policies, core boundary leaks) that
# `--profile=pr` deliberately omits. No `--changed-since`: this gate
# exists to see the integrated tree.
args: --profile=full
# Pure pass/fail gate — the weekly sweep files the tracking issues.
# (No `autofix` input: homeboy-action@v2 has none, so passing one is
# dead configuration. This gate mutates nothing regardless.)
auto-issue: 'false'
# BLOCKING. #10557's defect was a gate that reported success while
# measuring nothing, so the one property this job enforces unconditionally
# is that the audit had a corpus at all.
#
# The assertion lives in .github/assert-audit-corpus.sh, not inline here,
# because an inline `run:` block can only ever be checked by a YAML
# substring test — and a substring test is exactly what let this step ship
# broken twice (wrong result filename, then a wrong jq path bolted on top
# of it). The script is executed against recorded fixtures by
# tests/audit_debt_workflow_test.rs, including one fixture per historical
# defect, so the assertion is verified by running it rather than by
# reading it.
#
# `files_scanned` is the CONVENTION corpus (index files and single-file
# directories excluded — see #10558). It is a lower bound on what the
# audit actually read, which is exactly what a "did this run at all"
# assertion wants. `review audit` itself already errors when the wider
# source-policy corpus is empty, so the two checks bracket the failure.
- name: Assert the audit actually scanned the tree
run: bash .github/assert-audit-corpus.sh
- name: Report the (non-blocking) audit findings verdict
if: steps.audit.outcome == 'failure'
run: |
set -euo pipefail
result="${HOMEBOY_OUTPUT_DIR:-}/review-audit.json"
echo "::warning::Full-profile audit reported findings. This verdict is REPORTING-ONLY until #10569 regenerates the baseline; the corpus assertion above is the blocking part. Expand the audit group in this job's log for the finding list."
# Put the size of the debt in the job summary. A warning annotation
# nobody counts is how a reporting-only verdict quietly becomes
# permanent; a number that has to be looked at every merge is not.
{
echo "### Full-tree audit: reporting-only verdict (#10569)"
echo
if [ -s "${result}" ]; then
jq -r '"- corpus: \(.data.summary.files_scanned) file(s)",
"- total findings: \(.data.findings | length)",
"- new since baseline: \(.data.baseline_comparison.new_items | length)",
"- resolved since baseline: \(.data.baseline_comparison.resolved_fingerprints | length)"' \
"${result}" 2>/dev/null \
|| echo "- could not read \`${result}\`"
else
echo "- no audit result at \`${result}\` (see the corpus assertion step)"
fi
} >> "${GITHUB_STEP_SUMMARY}"
# ── Release-blocking suite, at full scope, on the merged result ──
# `review lint` + `review test` is exactly release.yml's RELEASE_BLOCKING_COMMANDS
# (default 'review lint,review test'). Split into two jobs, and `review test`
# carries `--skip-lint`, both matching release.yml's gate-lint / gate-test
# split so lint work is not duplicated.
#
# The ONE deliberate difference from the release gates: no `--changed-since`.
# The release passes `--changed-since ${{ github.event.before }}`; omitting it
# lets `scope: auto` resolve to full scope on push. Matching the release's
# narrowing here would reproduce the blind spot instead of closing it.
#
# No `differential-gating` (it only applies to pull_request events anyway) and
# no `autofix` input — homeboy-action@v2 has no such input, so there is no
# autofix/auto-push path to disable. This gate never mutates the repo.
full-lint-gate:
name: Full-suite lint gate
if: github.event_name == 'push'
needs: gate-build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Download homeboy binary
uses: actions/download-artifact@v4
with:
name: homeboy-binary
path: .homeboy-bin
- name: Verify homeboy binary present
run: |
if [ ! -f .homeboy-bin/homeboy ]; then
echo "::error::Build artifact missing from upstream Build job: .homeboy-bin/homeboy was not produced/uploaded by gate-build. This is a CI artifact-handoff problem, not a lint finding on main. Re-run the failed Build job or investigate the homeboy-binary upload step." >&2
exit 1
fi
chmod +x .homeboy-bin/homeboy
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v3
continue-on-error: true
with:
app-id: ${{ secrets.HOMEBOY_APP_ID }}
private-key: ${{ secrets.HOMEBOY_APP_PRIVATE_KEY }}
- uses: Extra-Chill/homeboy-action@v2
with:
binary-path: .homeboy-bin/homeboy
commands: review lint
expected-commands: review audit,review lint,review test
# Durable notification. The failing check is the primary signal, but
# nobody watches main's commit list; the issue survives the run.
auto-issue: 'true'
app-token: ${{ steps.app-token.outputs.token || '' }}
- name: Attribute failure to this merge
if: failure()
env:
MERGE_SHA: ${{ github.sha }}
MERGE_BEFORE: ${{ github.event.before }}
MERGE_SUBJECT: ${{ github.event.head_commit.message }}
MERGE_AUTHOR: ${{ github.event.head_commit.author.name }}
GATE: lint
run: bash .github/report-red-main.sh
full-test-gate:
name: Full-suite test gate
if: github.event_name == 'push'
needs: gate-build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Download homeboy binary
uses: actions/download-artifact@v4
with:
name: homeboy-binary
path: .homeboy-bin
- name: Verify homeboy binary present
run: |
if [ ! -f .homeboy-bin/homeboy ]; then
echo "::error::Build artifact missing from upstream Build job: .homeboy-bin/homeboy was not produced/uploaded by gate-build. This is a CI artifact-handoff problem, not a test finding on main. Re-run the failed Build job or investigate the homeboy-binary upload step." >&2
exit 1
fi
chmod +x .homeboy-bin/homeboy
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v3
continue-on-error: true
with:
app-id: ${{ secrets.HOMEBOY_APP_ID }}
private-key: ${{ secrets.HOMEBOY_APP_PRIVATE_KEY }}
- uses: Extra-Chill/homeboy-action@v2
with:
binary-path: .homeboy-bin/homeboy
commands: review test
expected-commands: review audit,review lint,review test
# Matches release.yml gate-test: lint is its own job above.
args: --skip-lint
auto-issue: 'true'
app-token: ${{ steps.app-token.outputs.token || '' }}
- name: Attribute failure to this merge
if: failure()
env:
MERGE_SHA: ${{ github.sha }}
MERGE_BEFORE: ${{ github.event.before }}
MERGE_SUBJECT: ${{ github.event.head_commit.message }}
MERGE_AUTHOR: ${{ github.event.head_commit.author.name }}
GATE: test
run: bash .github/report-red-main.sh
full-audit:
name: Full-tree audit → tracking issues
if: github.event_name != 'push'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Check out Homeboy Action
uses: actions/checkout@v6
with:
repository: Extra-Chill/homeboy-action
ref: v2
path: .homeboy-action
- name: Generate GitHub App token
id: app-token
uses: actions/create-github-app-token@v3
continue-on-error: true
with:
app-id: ${{ secrets.HOMEBOY_APP_ID }}
private-key: ${{ secrets.HOMEBOY_APP_PRIVATE_KEY }}
- uses: ./.homeboy-action
with:
source: .
component: homeboy
commands: review audit
# Full profile enables the whole-tree discovery detectors. No
# --changed-since: this is a deliberate repo-wide sweep.
args: --profile=${{ github.event.inputs.profile || 'full' }}
# File one deduplicated tracking issue per finding kind. The action
# auto-enables this on non-PR events, but set it explicitly so the
# intent of this workflow is unambiguous.
auto-issue: 'true'
# Advisory sweep — never pushes. Findings become the roadmap; fixes
# land through their own reviewed PRs. (There is no `autofix` input on
# homeboy-action@v2, so the `autofix: 'false'` that used to sit here
# was dead configuration asserting a property it did not enforce.)
app-token: ${{ steps.app-token.outputs.token || '' }}