ci: audit example with-keys against current action.yml input contracts (#136)

eFAILution · web-flow · commit 106bfbd724d5 · 2026-05-07T19:22:57.000-04:00
* ci: audit example with-keys against current action.yml input contracts Closes the gap that let the recent example-staleness drift through unnoticed. Composite-action consumers passing ``with:`` keys that aren't declared in ``action.yml::inputs`` see a runtime warning, not a failure. CI never *runs* the example workflows — only YAML-validates them — so a contract trim could silently leave ``examples/`` referencing inputs the action no longer accepts. PR #134 fixed the symptom; this PR closes the source. What ships ========== * ``scripts/ci/check_example_inputs.py`` — walks every ``examples/**/*.yml``, finds each ``uses: huntridge-labs/argus/.github/actions/<name>@<ref>`` step, and diffs its ``with:`` keys against the matching action's declared inputs. Reports unknown keys with job/step/file location. Exits 1 on any miss, 0 on clean, 2 on internal error (missing actions dir). Surfaces: --json machine-readable output --gh-annotations ``::error file=...:: PR-inline rendering --paths <dir>... override the default examples roots --actions-dir <p> override the default actions dir * ``tests/unit/test_check_example_inputs.py`` — 18 tests covering: input-name collection, clean / unknown-input / multi-key / missing-action / yaml-parse-error workflows, non-argus uses-step ignored, exit codes (0/1/2), JSON output shape, gh-annotations stderr emission, and a smoke test that the actual repo's ``examples/`` passes today. * ``.github/workflows/test-examples-functional.yml`` — new ``audit-action-inputs`` job runs once across the whole example tree (cross-action correlation, not per-example). Wired into the ``examples-summary`` aggregator so any failure blocks the workflow. Uses env-var-injected ``needs.<>.result`` values to stay workflow-injection-safe per the GitHub security guidance. Why a single job, not the per-example matrix ============================================= The audit needs the full ``action.yml`` input contract for every referenced action plus every ``with:`` block across the tree. Splitting per file would re-load the contracts N times and fragment the failure summary. One job, one parse pass, one report — fast enough that the matrix split adds nothing. Local verification ================== * ``python -m scripts.ci.check_example_inputs`` → exit 0 on ``feat/argus-portability`` post PRs #134 / #135. * Full SDK suite + the new tests: 2911 passed, 2 skipped, 7 deselected. * ``test-examples-functional.yml`` parses as valid YAML. Future ====== If a future contract trim lands without an examples follow-up, the audit fails the next PR-validation run with an inline annotation pointing at the offending example. Either fix the example, or update the action's contract — see the script's docstring for the resolution flow. * fix(ci): silence check_version_refs false-positives in audit script + tests The new audit script and its tests reference ``huntridge-labs/argus/. github/actions/<name>@<ref>`` as a literal placeholder pattern in docstrings, comments, and a test-helper f-string. The ``check_version_refs.py`` gate parses that as a real version ref and flags it as uncovered. Three minimal fixes — keep the placeholder text intact where the docstring needed it, just teach the gate to skip those lines: * scripts/ci/check_example_inputs.py:61 — added the ``release-it- ignore`` IGNORE_MARKER suffix to the comment line that documents the regex shape. * tests/unit/test_check_example_inputs.py:4 — embedded the marker inline in the docstring's prose. The marker is a plain string the gate searches for, so prose context works fine. * tests/unit/test_check_example_inputs.py::_write_example — pulled ``huntridge-labs/argus/.github/actions/`` into a local variable so the literal pattern doesn't appear on the same source line as the action name + ref. The rendered YAML still has a single ``uses:`` line, which is what the audit-under-test consumes. Verified: ``check_version_refs`` exits 0 against tracked files, the audit script + 18 unit tests still pass. --------- Co-authored-by: eFAILution <eFAILution@users.noreply.github.com>
diff --git a/.github/workflows/test-examples-functional.yml b/.github/workflows/test-examples-functional.yml
@@ -209,6 +209,55 @@ jobs:
             fi
           done
 
+  # ============================================================================
+  # Audit example workflows against current action.yml input contracts
+  # Runs once across the whole tree (cross-action correlation, not
+  # per-example). Catches the bug class where a contract trim leaves
+  # documented examples passing inputs the action no longer accepts —
+  # composite actions only emit a runtime warning for those, so the
+  # drift never surfaces in the syntax-only validate-examples matrix.
+  # See scripts/ci/check_example_inputs.py for the audit logic.
+  # ============================================================================
+  audit-action-inputs:
+    name: Audit Example Action Inputs
+    runs-on: ubuntu-latest
+    needs: discover-examples
+    timeout-minutes: 5
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+
+      - name: Install PyYAML
+        run: pip install PyYAML
+
+      - name: Audit example with-keys against action.yml inputs
+        run: |
+          # --gh-annotations emits ::error file=...:: lines that render
+          # inline in the PR Files-changed view, pinpointing the
+          # offending example.
+          python -m scripts.ci.check_example_inputs --gh-annotations
+
+      - name: Summarize
+        if: always()
+        run: |
+          {
+            echo "## Example Action-Input Audit"
+            echo ""
+            echo "Checks every \`uses: huntridge-labs/argus/.github/actions/<name>@<ref>\` step in"
+            echo "\`examples/**\` against its action's current \`action.yml::inputs\` list."
+            echo ""
+            echo "Source of truth: \`.github/actions/<name>/action.yml\`."
+            echo "Audit script: \`scripts/ci/check_example_inputs.py\`."
+            echo ""
+          } >> "$GITHUB_STEP_SUMMARY"
+
   # ============================================================================
   # Final Summary
   # ============================================================================
@@ -218,31 +267,42 @@ jobs:
     needs:
       - validate-examples
       - validate-configs
+      - audit-action-inputs
     if: always()
 
     steps:
       - name: Generate Summary
+        env:
+          NEEDS_VALIDATE_EXAMPLES_RESULT: ${{ needs.validate-examples.result }}
+          NEEDS_VALIDATE_CONFIGS_RESULT: ${{ needs.validate-configs.result }}
+          NEEDS_AUDIT_ACTION_INPUTS_RESULT: ${{ needs.audit-action-inputs.result }}
         run: |
+          examples_status="❌ Failed"
+          [ "$NEEDS_VALIDATE_EXAMPLES_RESULT" = "success" ] && examples_status="✅ Passed"
+          configs_status="❌ Failed"
+          [ "$NEEDS_VALIDATE_CONFIGS_RESULT" = "success" ] && configs_status="✅ Passed"
+          audit_status="❌ Failed"
+          [ "$NEEDS_AUDIT_ACTION_INPUTS_RESULT" = "success" ] && audit_status="✅ Passed"
+
           {
             echo "# Example Validation Results"
             echo ""
             echo "| Validation Type | Status |"
             echo "|-----------------|--------|"
-            echo "| Workflow Examples | ${{ needs.validate-examples.result == 'success' && '✅ Passed' || '❌ Failed' }} |"
-            echo "| Config Examples | ${{ needs.validate-configs.result == 'success' && '✅ Passed' || '❌ Failed' }} |"
+            echo "| Workflow Examples | $examples_status |"
+            echo "| Config Examples | $configs_status |"
+            echo "| Action-Input Audit | $audit_status |"
             echo ""
             echo "> **Note**: Functional testing of actions is handled by \`test-actions.yml\`"
             echo "> This workflow focuses on validating example **documentation** is correct."
             echo ""
           } >> "$GITHUB_STEP_SUMMARY"
 
-          if [[ "${NEEDS_VALIDATE_EXAMPLES_RESULT}" == "success" && \
-                "${NEEDS_VALIDATE_CONFIGS_RESULT}" == "success" ]]; then
+          if [ "$NEEDS_VALIDATE_EXAMPLES_RESULT" = "success" ] && \
+             [ "$NEEDS_VALIDATE_CONFIGS_RESULT" = "success" ] && \
+             [ "$NEEDS_AUDIT_ACTION_INPUTS_RESULT" = "success" ]; then
             echo "✅ **All examples validated successfully!**" >> "$GITHUB_STEP_SUMMARY"
           else
             echo "❌ **Some validations failed - review above for details**" >> "$GITHUB_STEP_SUMMARY"
             exit 1
           fi
-        env:
-          NEEDS_VALIDATE_EXAMPLES_RESULT: ${{ needs.validate-examples.result }}
-          NEEDS_VALIDATE_CONFIGS_RESULT: ${{ needs.validate-configs.result }}
diff --git a/scripts/ci/check_example_inputs.py b/scripts/ci/check_example_inputs.py
@@ -0,0 +1,268 @@
+#!/usr/bin/env python3
+"""Audit example workflows against current action.yml input contracts.
+
+Why this exists
+===============
+
+Composite-action consumers passing inputs that aren't declared in
+``action.yml::inputs`` see a runtime warning, not a failure. CI never
+*runs* the example workflows — it only validates their YAML syntax.
+That gap let a pre-1.0.0 contract trim drift through unnoticed: every
+``examples/workflows/actions-scanner-zap-*.yml`` and several
+``examples/github-enterprise/*.yml`` kept passing inputs that
+``scanner-zap`` / ``scanner-container`` / ``scanner-gitleaks`` had
+already removed. Surfaced during an external GHES consumer migration
+audit and fixed in PR #134.
+
+This script closes the gap: it parses every action's ``action.yml``
+to build the source-of-truth input list, walks every example
+workflow's ``with:`` blocks for ``uses: huntridge-labs/argus/.github/
+actions/<name>@<ref>`` steps, and reports any ``with:`` key that
+isn't in the action's current contract. Run from CI, this prevents
+the next contract trim from silently breaking the docs.
+
+Usage
+=====
+
+    python -m scripts.ci.check_example_inputs              # all examples
+    python -m scripts.ci.check_example_inputs --paths examples/workflows
+    python -m scripts.ci.check_example_inputs --json       # machine-readable
+
+Exit codes
+==========
+
+* 0 — every ``with:`` key is declared in the matching action.yml
+* 1 — at least one unknown key found (failure list printed)
+* 2 — internal error (couldn't parse an action.yml, etc.)
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+from typing import Iterator
+
+import yaml
+
+
+# Composite actions live here. Source of truth for input names.
+ARGUS_ACTIONS_DIR = Path(".github/actions")
+
+# Examples whose with: blocks we audit. Two roots cover all canonical
+# consumer-facing patterns:
+#   examples/workflows/         — composite-action-based examples
+#   examples/github-enterprise/ — GHES-targeted examples
+DEFAULT_EXAMPLE_ROOTS = ("examples/workflows", "examples/github-enterprise")
+
+# Action references look like
+#   uses: huntridge-labs/argus/.github/actions/<name>@<ref>  # release-it-ignore
+# Ref can be a tag, branch, or SHA — we don't care which.
+_ACTION_USES_RE = re.compile(
+    r"^huntridge-labs/argus/\.github/actions/([^@]+)@"
+)
+
+
+def collect_action_inputs(actions_dir: Path) -> dict[str, set[str]]:
+    """Return ``{action_name: {input_name, ...}}`` for every action.yml.
+
+    Missing or unparsable ``action.yml`` files are reported but do not
+    halt the walk — they'll surface as "missing action" errors when an
+    example references them.
+    """
+    out: dict[str, set[str]] = {}
+    for action_dir in sorted(actions_dir.iterdir()):
+        yml = action_dir / "action.yml"
+        if not yml.is_file():
+            continue
+        try:
+            data = yaml.safe_load(yml.read_text()) or {}
+        except yaml.YAMLError as exc:
+            # Surface but don't halt — a broken action.yml is a
+            # different bug class.
+            print(
+                f"::warning file={yml}::failed to parse action.yml: {exc}",
+                file=sys.stderr,
+            )
+            continue
+        out[action_dir.name] = set((data.get("inputs") or {}).keys())
+    return out
+
+
+def iter_example_workflows(roots: tuple[str, ...]) -> Iterator[Path]:
+    """Yield every ``*.yml`` under the given example roots."""
+    for root in roots:
+        root_path = Path(root)
+        if not root_path.is_dir():
+            continue
+        yield from sorted(root_path.rglob("*.yml"))
+
+
+def audit_workflow(
+    wf: Path,
+    action_inputs: dict[str, set[str]],
+) -> list[dict]:
+    """Return one issue dict per problem in *wf*.
+
+    Each issue has ``file``, ``job``, ``step``, ``action``, ``kind``,
+    and either ``unknown`` (a sorted list of unrecognized keys) or
+    ``message`` (free text for missing-action / parse-fail cases).
+    """
+    issues: list[dict] = []
+    try:
+        data = yaml.safe_load(wf.read_text())
+    except yaml.YAMLError as exc:
+        issues.append({
+            "file": str(wf),
+            "job": None,
+            "step": None,
+            "action": None,
+            "kind": "yaml_parse_error",
+            "message": str(exc),
+        })
+        return issues
+
+    if not isinstance(data, dict):
+        return issues
+    jobs = data.get("jobs") or {}
+    if not isinstance(jobs, dict):
+        return issues
+
+    for job_name, job in jobs.items():
+        if not isinstance(job, dict):
+            continue
+        for step in (job.get("steps") or []):
+            if not isinstance(step, dict):
+                continue
+            uses = step.get("uses", "")
+            if not isinstance(uses, str):
+                continue
+            m = _ACTION_USES_RE.match(uses)
+            if not m:
+                continue
+            action = m.group(1)
+            step_name = step.get("name", "?")
+
+            if action not in action_inputs:
+                issues.append({
+                    "file": str(wf),
+                    "job": job_name,
+                    "step": step_name,
+                    "action": action,
+                    "kind": "missing_action",
+                    "message": (
+                        f"action '{action}' is not present at "
+                        f"{ARGUS_ACTIONS_DIR / action}"
+                    ),
+                })
+                continue
+
+            with_keys = set((step.get("with") or {}).keys())
+            unknown = with_keys - action_inputs[action]
+            if unknown:
+                issues.append({
+                    "file": str(wf),
+                    "job": job_name,
+                    "step": step_name,
+                    "action": action,
+                    "kind": "unknown_input",
+                    "unknown": sorted(unknown),
+                })
+
+    return issues
+
+
+def format_human(issues: list[dict]) -> str:
+    if not issues:
+        return "✓ all example with: keys match current action.yml contracts\n"
+    lines = []
+    by_file: dict[str, list[dict]] = {}
+    for i in issues:
+        by_file.setdefault(i["file"], []).append(i)
+    for f, group in sorted(by_file.items()):
+        lines.append(f"\n{f}:")
+        for i in group:
+            loc = f"  {i['job'] or '?'}.{i['step'] or '?'}"
+            if i["kind"] == "unknown_input":
+                lines.append(
+                    f"{loc} (action: {i['action']}): unknown with: keys "
+                    f"{i['unknown']}"
+                )
+            elif i["kind"] == "missing_action":
+                lines.append(f"{loc}: {i['message']}")
+            else:
+                lines.append(f"{loc}: {i['kind']} — {i.get('message', '')}")
+    lines.append("")
+    lines.append(f"{len(issues)} issue(s) found across {len(by_file)} file(s)")
+    return "\n".join(lines) + "\n"
+
+
+def format_actions_annotations(issues: list[dict]) -> str:
+    """GitHub Actions ``::error file=...,line=...::message`` annotations."""
+    out = []
+    for i in issues:
+        if i["kind"] == "unknown_input":
+            msg = (
+                f"action '{i['action']}' does not declare these "
+                f"with: keys: {', '.join(i['unknown'])}. "
+                f"Compare against {ARGUS_ACTIONS_DIR / i['action']}/action.yml inputs."
+            )
+        elif i["kind"] == "missing_action":
+            msg = i["message"]
+        else:
+            msg = i.get("message", i["kind"])
+        out.append(f"::error file={i['file']}::{msg}")
+    return "\n".join(out) + ("\n" if out else "")
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        description=__doc__.split("\n\n", 1)[0],
+    )
+    parser.add_argument(
+        "--paths", nargs="+", default=list(DEFAULT_EXAMPLE_ROOTS),
+        help="Roots to walk for *.yml example files",
+    )
+    parser.add_argument(
+        "--json", action="store_true",
+        help="Emit issues as a JSON array on stdout",
+    )
+    parser.add_argument(
+        "--actions-dir", type=Path, default=ARGUS_ACTIONS_DIR,
+        help="Directory containing action.yml files",
+    )
+    parser.add_argument(
+        "--gh-annotations", action="store_true",
+        help="Also emit ``::error::`` annotations for GitHub Actions UI",
+    )
+    args = parser.parse_args(argv)
+
+    if not args.actions_dir.is_dir():
+        print(
+            f"actions directory not found: {args.actions_dir}",
+            file=sys.stderr,
+        )
+        return 2
+
+    action_inputs = collect_action_inputs(args.actions_dir)
+
+    issues: list[dict] = []
+    for wf in iter_example_workflows(tuple(args.paths)):
+        issues.extend(audit_workflow(wf, action_inputs))
+
+    if args.json:
+        json.dump(issues, sys.stdout, indent=2, sort_keys=True)
+        sys.stdout.write("\n")
+    else:
+        sys.stdout.write(format_human(issues))
+
+    if args.gh_annotations and issues:
+        sys.stderr.write(format_actions_annotations(issues))
+
+    return 1 if issues else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/tests/unit/test_check_example_inputs.py b/tests/unit/test_check_example_inputs.py
