ci: audit every blueprint on a schedule - #1049
Conversation
dosubot
Bot
added
the
size:XS
built with Refined Cloudflare Pages Action⚡ Cloudflare Pages Deployment
|
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow to periodically audit all blueprints with the existing validate-docker-compose.ts and validate-template.ts scripts, surfacing rule drift without impacting PR CI.
Changes:
- Introduces a scheduled (weekly) + manual (
workflow_dispatch) workflow to run both validators across everyblueprints/*/directory. - Generates a GitHub Actions job summary with failing counts and per-blueprint failure snippets.
- Adds an optional manual “gate” mode (
fail_on_findings) to fail the run when findings exist.
| fi | ||
|
|
||
| - name: Fail if requested | ||
| if: inputs.fail_on_findings && (steps.audit.outputs.compose_failed != '0' || steps.audit.outputs.toml_failed != '0') |
| COMPOSE_FAILED="" | ||
| TOML_FAILED="" | ||
|
|
|
Both correct and both fixed in 1. if: >-
github.event_name == 'workflow_dispatch' &&
github.event.inputs.fail_on_findings == 'true' &&
(steps.audit.outputs.compose_failed != '0' || steps.audit.outputs.toml_failed != '0')Gated on the event first, and the value read through 2. Fixed Now uses WORK="${RUNNER_TEMP:-/tmp}"
OUT="$WORK/validator-out.txt"
REPORT="$WORK/audit-report.md"
SUMMARY="$WORK/audit-summary.md"
rm -f "$OUT" "$REPORT" "$SUMMARY"Verified rather than assumed. Re-extracted the Same counts as before (and still matching the hand-derived numbers in #1047), summary still renders 110 finding sections, and the planted file is gone rather than appended. I also confirmed the previous version would have leaked it — the old script |
| if [ -f "$REPORT" ]; then | ||
| cat "$REPORT" >> "$SUMMARY" | ||
| else | ||
| echo "No findings." >> "$SUMMARY" | ||
| fi | ||
|
|
||
| cat "$SUMMARY" >> "$GITHUB_STEP_SUMMARY" |
|
Good catch — fixed in I measured it before deciding how to respond, because the answer changes the fix: the summary is currently 35 KB against a ~1 MB cap, so there's roughly 30× headroom and it would take about 3,200 finding sections to hit it today. Not urgent — but the failure mode you describe is the bad kind, because it gets worse as the backlog grows and it truncates silently, so the run would look healthy while hiding exactly the findings the job exists to report. Two changes:
Verified both branches by executing the step, not by reading it. Truncation path, forced with a 2,000-byte limit: Normal path, real 512 KB limit against a representative sample: One note on scope: I verified the untruncated branch against a 5-blueprint sample (3 known-failing, 2 known-passing) rather than the full 500, because my local runs of the complete sweep kept getting killed. The full sweep was verified in the previous round and produced the same |
| if [ -f "$dir/docker-compose.yml" ]; then | ||
| if ! (cd build-scripts && pnpm exec tsx validate-docker-compose.ts \ | ||
| --file "../$dir/docker-compose.yml") > "$OUT" 2>&1; then | ||
| COMPOSE_FAILED="$COMPOSE_FAILED $id" | ||
| { | ||
| echo "### \`$id\` — docker-compose.yml" | ||
| echo '```' | ||
| grep '❌' "$OUT" || cat "$OUT" | ||
| echo '```' | ||
| } >> "$REPORT" | ||
| fi | ||
| fi | ||
|
|
||
| if [ -f "$dir/template.toml" ]; then | ||
| if ! (cd build-scripts && pnpm exec tsx validate-template.ts \ | ||
| --dir "../$dir") > "$OUT" 2>&1; then | ||
| TOML_FAILED="$TOML_FAILED $id" | ||
| { | ||
| echo "### \`$id\` — template.toml" | ||
| echo '```' | ||
| grep '❌' "$OUT" || cat "$OUT" | ||
| echo '```' | ||
| } >> "$REPORT" | ||
| fi | ||
| fi |
| if [ "$SUMMARY_BYTES" -gt "$SUMMARY_LIMIT" ]; then | ||
| head -c "$SUMMARY_LIMIT" "$SUMMARY" >> "$GITHUB_STEP_SUMMARY" | ||
| { | ||
| echo "" | ||
| echo "---" | ||
| echo "" | ||
| echo "_Report truncated at $SUMMARY_LIMIT bytes (full size: $SUMMARY_BYTES)._" | ||
| echo "_Download the \`blueprint-audit\` artifact for the complete output._" | ||
| } >> "$GITHUB_STEP_SUMMARY" | ||
| else | ||
| cat "$SUMMARY" >> "$GITHUB_STEP_SUMMARY" | ||
| fi |
|
Both right, both fixed in 1. Skipping validators when a file is missing. Agreed, and the reasoning is stronger than "consistency" — the skip hides exactly the blueprints most likely to be broken. Confirmed the validators do the right thing on their own: Both Previously both would have been silently clean. Re-ran the same 5 real blueprints afterwards and got the unchanged 2. That renders as literal text inside an open fence — the truncation would have been invisible, which defeats the point of the warning. Took your suggestion: on overflow, emit only the header table plus the notice, and rely on the artifact. Verified the header is exactly 7 lines (line 8 is the first Also reworded from "truncated" to "omitted", since the report is no longer partially included — it is entirely in the artifact. |
What
Adds a scheduled workflow that runs
validate-docker-compose.tsandvalidate-template.tsover every blueprint, so rule drift gets reported instead of accumulating silently.Implements the CI suggestion from #1047.
Why
The per-PR validators only look at blueprints a PR touches — which is the right call for CI cost, but it means a template is checked when it lands and then never again. When a rule is added or tightened, every existing blueprint that breaks it stays broken with nothing surfacing it.
That is not hypothetical. Running the validators across all 500 today:
validate-docker-compose.tsvalidate-template.tsAnd #1048 demonstrated the mechanism live: editing
vault/docker-compose.ymlmade CI look atvault/template.tomlfor the first time since a rule tightened, and it failed on a problem that had nothing to do with that PR.Design decisions
Scheduled + manual only, never on
pull_request. A contributor must not be blocked by a pre-existing failure they did not introduce. That is exactly what happened on #1048, and it should not be the default experience.Reports rather than fails. The default run emits a
::warning::and writes a full breakdown to the job summary.workflow_dispatchtakes afail_on_findingsboolean for when you want it to gate — so you can flip it on once the backlog is cleared, without editing the workflow.Reuses the existing validators unchanged. No new checking logic to keep in sync; it runs the same two scripts the PR workflows do, with the same
pnpm exec tsxinvocation and pinned Node 20.16.0 / pnpm 8 setup asvalidate-docker-compose.yml.permissions: contents: read— it only reads.The job summary it produces
Ran the workflow's audit step verbatim against the current tree. It renders a summary table plus one section per finding, with the validator's real output:
110 finding sections in total.
Testing
I extracted the
run:block from the workflow and executed it as a shell script against the real repo, rather than eyeballing the YAML:Those numbers match what I counted by hand when filing #1047, which is the check that matters — the script isn't just exiting cleanly, it is finding the right things. The YAML also parses (triggers
schedule+workflow_dispatch, 7 steps).One caveat I want to be straight about. My local pnpm is broken (
corepacktries to reinstall on everypnpm execand errors), so my local run substitutednpx tsxforpnpm exec tsx. The committed workflow keepspnpm execto matchvalidate-docker-compose.yml, and the surrounding setup steps are copied from that workflow verbatim — but thepnpm execpath specifically is the one line I verified by consistency with the existing working workflow rather than by executing it myself. Worth a look on the first scheduled run.Note on the current findings
To be clear, I am not proposing to fix the 110 in this PR. Some are almost certainly intentional — the port-mapping findings are
adguardhome(53/udp DNS),poste.io(SMTP/POP/IMAP),wg-easy(51820/udp),enshrouded(game server), all of which genuinely need host ports. Those may argue for an allowance in the validator rather than a change to the templates. This PR just makes the number visible.