ci: audit example with-keys against current action.yml input contracts#136
Merged
eFAILution merged 2 commits intofeat/argus-portabilityfrom May 7, 2026
Merged
Conversation
Closes the gap that let the recent example-staleness drift through unnoticed. Composite-action consumers passing ``with:`` keys that aren't declared in ``action.yml::inputs`` see a runtime warning, not a failure. CI never *runs* the example workflows — only YAML-validates them — so a contract trim could silently leave ``examples/`` referencing inputs the action no longer accepts. PR #134 fixed the symptom; this PR closes the source. What ships ========== * ``scripts/ci/check_example_inputs.py`` — walks every ``examples/**/*.yml``, finds each ``uses: huntridge-labs/argus/.github/actions/<name>@<ref>`` step, and diffs its ``with:`` keys against the matching action's declared inputs. Reports unknown keys with job/step/file location. Exits 1 on any miss, 0 on clean, 2 on internal error (missing actions dir). Surfaces: --json machine-readable output --gh-annotations ``::error file=...:: PR-inline rendering --paths <dir>... override the default examples roots --actions-dir <p> override the default actions dir * ``tests/unit/test_check_example_inputs.py`` — 18 tests covering: input-name collection, clean / unknown-input / multi-key / missing-action / yaml-parse-error workflows, non-argus uses-step ignored, exit codes (0/1/2), JSON output shape, gh-annotations stderr emission, and a smoke test that the actual repo's ``examples/`` passes today. * ``.github/workflows/test-examples-functional.yml`` — new ``audit-action-inputs`` job runs once across the whole example tree (cross-action correlation, not per-example). Wired into the ``examples-summary`` aggregator so any failure blocks the workflow. Uses env-var-injected ``needs.<>.result`` values to stay workflow-injection-safe per the GitHub security guidance. Why a single job, not the per-example matrix ============================================= The audit needs the full ``action.yml`` input contract for every referenced action plus every ``with:`` block across the tree. Splitting per file would re-load the contracts N times and fragment the failure summary. One job, one parse pass, one report — fast enough that the matrix split adds nothing. Local verification ================== * ``python -m scripts.ci.check_example_inputs`` → exit 0 on ``feat/argus-portability`` post PRs #134 / #135. * Full SDK suite + the new tests: 2911 passed, 2 skipped, 7 deselected. * ``test-examples-functional.yml`` parses as valid YAML. Future ====== If a future contract trim lands without an examples follow-up, the audit fails the next PR-validation run with an inline annotation pointing at the offending example. Either fix the example, or update the action's contract — see the script's docstring for the resolution flow.
Contributor
E2E Test Coverage Report
Summary
✅ All Actions Have E2E Coverage |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
… tests The new audit script and its tests reference ``huntridge-labs/argus/. github/actions/<name>@<ref>`` as a literal placeholder pattern in docstrings, comments, and a test-helper f-string. The ``check_version_refs.py`` gate parses that as a real version ref and flags it as uncovered. Three minimal fixes — keep the placeholder text intact where the docstring needed it, just teach the gate to skip those lines: * scripts/ci/check_example_inputs.py:61 — added the ``release-it- ignore`` IGNORE_MARKER suffix to the comment line that documents the regex shape. * tests/unit/test_check_example_inputs.py:4 — embedded the marker inline in the docstring's prose. The marker is a plain string the gate searches for, so prose context works fine. * tests/unit/test_check_example_inputs.py::_write_example — pulled ``huntridge-labs/argus/.github/actions/`` into a local variable so the literal pattern doesn't appear on the same source line as the action name + ref. The rendered YAML still has a single ``uses:`` line, which is what the audit-under-test consumes. Verified: ``check_version_refs`` exits 0 against tracked files, the audit script + 18 unit tests still pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes the audit gap that let the recent example-staleness drift through unnoticed (PRs #134 / #135 fixed the symptom; this PR closes the source).
Composite-action consumers passing
with:keys not declared inaction.yml::inputssee a runtime warning, not a failure. CI never runs the example workflows — only YAML-validates them — so a contract trim could silently leaveexamples/referencing inputs the action no longer accepts.Changes Made
Details
scripts/ci/check_example_inputs.py(new) — walks everyexamples/**/*.yml, finds eachuses: huntridge-labs/argus/.github/actions/<name>@<ref>step, diffs itswith:keys against the matching action's declared inputs. Reports unknown keys with job/step/file location.examples/workflows/andexamples/github-enterprise/--paths <dir>...--actions-dir <p>--json--gh-annotations::error file=...::lines that render inline in PR Files-changed viewExit codes:
0clean,1unknown-key found,2internal error.tests/unit/test_check_example_inputs.py(new) — 18 tests covering:uses:steps ignored--gh-annotationsstderr emissionexamples/passes today).github/workflows/test-examples-functional.yml(modified) — newaudit-action-inputsjob runs once across the whole example tree (cross-action correlation, not per-example). Wired into theexamples-summaryaggregator so any failure blocks the workflow. Uses env-var-injectedneeds.<>.resultvalues to stay workflow-injection-safe per GitHub's security guidance.Why a single job, not the per-example matrix
The audit needs the full
action.ymlinput contract for every referenced action plus everywith:block across the tree. Splitting per file would re-load the contracts N times and fragment the failure summary. One job, one parse pass, one report.Cross-platform safety
needs.<>.resultvalues flow throughenv:rather than being interpolated insiderun:script bodies.Testing
Test Results
Security Considerations
(The new workflow job follows GitHub's workflow-injection-safety guidance:
${{ needs.<>.result }}flows throughenv:vars, not insiderun:script bodies.)AI Context Updates (.ai/)
.ai/architecture.yamlupdated.ai/workflows.yamlupdated.ai/decisions.yamlupdated.ai/errors.yamlupdated(Pure CI infra add. The script's docstring documents the resolution flow for future contributors.)
Checklist
Related Issues
N/A — closes the audit-gap follow-up tracked in PR #135's roadmap entry. Companion to PR #134 (which fixed the symptom).