|
| 1 | +# Argus Testing Strategy |
| 2 | + |
| 3 | +## Ref Lifecycle |
| 4 | + |
| 5 | +Version references flow through three stages, each automatically managed: |
| 6 | + |
| 7 | +### PR Branch (feature development) |
| 8 | +- Composite actions: `./` (local path — tests the PR's code) |
| 9 | +- Reusable workflows: `./` (local path via workflow_call) |
| 10 | +- `test-actions.yml` E2E: uses local checkout, validates the PR branch |
| 11 | +- `build-containers.yml`: builds images from the PR, scans with argus CLI from the PR |
| 12 | +- No version tags involved — everything is local |
| 13 | + |
| 14 | +### Main Branch (after merge) |
| 15 | +- Composite actions: still `./` for internal CI |
| 16 | +- Reusable workflows: callers outside the repo use `@main` |
| 17 | +- `security-scan.yml`: runs argus scan against main (scheduled weekly + on push) |
| 18 | +- All tests re-run on push to main as final validation |
| 19 | + |
| 20 | +### Release Tag (cut by release-it) |
| 21 | +- `release-it` bumps all version refs across the repo: |
| 22 | + - `version.yaml` (source of truth) |
| 23 | + - `package.json` `"version":` |
| 24 | + - `argus/__init__.py` `__version__ =` |
| 25 | + - `argus/containers.py` GHCR image tags |
| 26 | + - All `@X.Y.Z` refs in actions, workflows, examples, docs |
| 27 | +- Container images published to GHCR with version tag + `latest` |
| 28 | +- GitHub Release created with changelog |
| 29 | + |
| 30 | +## In-Repo Testing (PR Pipeline) |
| 31 | + |
| 32 | +All validation happens within the argus repo. No external repos required for PR gating. |
| 33 | + |
| 34 | +### Test Matrix |
| 35 | + |
| 36 | +| Workflow | What It Tests | Trigger | |
| 37 | +|----------|--------------|---------| |
| 38 | +| `test-unit.yml` | pytest suite (403+ tests), Python 3.11/3.12/3.13, version ref coverage | PR, push to main | |
| 39 | +| `test-actions.yml` | Composite action E2E with real scanners against test fixtures | PR (path-filtered), push to main | |
| 40 | +| `build-containers.yml` | Build images → scan with Trivy+Grype → test argus CLI → validate audit trail | PR (path-filtered) | |
| 41 | +| `test-examples-functional.yml` | Example workflow YAML validation | PR (path-filtered) | |
| 42 | +| `test-lint-workflows.yml` | actionlint on all workflow files | PR, push to main | |
| 43 | +| `security-scan.yml` | Dogfood: argus scans argus | push to main, weekly schedule | |
| 44 | + |
| 45 | +### Coverage Guarantees |
| 46 | + |
| 47 | +- **Unit tests**: 80%+ coverage enforced via pytest-cov |
| 48 | +- **Composite actions**: Every scanner action has E2E test in test-actions.yml |
| 49 | +- **Argus CLI**: build-containers.yml tests --list, scan, output validation, SARIF structure, audit trail |
| 50 | +- **Container images**: Built, scanned with Trivy+Grype, results posted as PR comment |
| 51 | +- **Version refs**: check-version-refs.py runs in CI, fails if any ref would be stale after release |
| 52 | + |
| 53 | +## External Validation: argus-test |
| 54 | + |
| 55 | +The `huntridge-labs/argus-test` repo serves as an **external consumer validation** that runs on a schedule against the latest published release. It does NOT gate PRs. |
| 56 | + |
| 57 | +### Purpose |
| 58 | +- Validates that published releases work from a consumer's perspective |
| 59 | +- Catches regressions that in-repo tests miss (real-world usage patterns) |
| 60 | +- Generates a dashboard with historical pass/fail rates |
| 61 | +- Alerts the team when the latest release breaks |
| 62 | + |
| 63 | +### What argus-test Needs (TODO) |
| 64 | + |
| 65 | +1. **Update to 1.0.0 interfaces**: The existing test workflows call `container-scan.yml`, `infrastructure-scan.yml`, etc. These were restored with the same interfaces but now use argus CLI internally. Tests should still pass. |
| 66 | + |
| 67 | +2. **Add argus CLI tests**: New test category that validates: |
| 68 | + - `argus scan --list` returns all scanners |
| 69 | + - `argus scan bandit --path <test-fixtures>` produces expected findings |
| 70 | + - `argus scan container --image nginx:1.19.0` detects known CVEs |
| 71 | + - `argus collect` merges multi-scanner results |
| 72 | + - `argus-audit.json` is produced with valid structure |
| 73 | + |
| 74 | +3. **Add SDK unit tests**: Run `pytest argus/tests/` against the published ref to ensure the SDK works when consumed via checkout (before PyPI is available). |
| 75 | + |
| 76 | +4. **Alerting**: When the scheduled run fails: |
| 77 | + - Open an issue on `huntridge-labs/argus` with the failure details |
| 78 | + - Include: which tests failed, the argus ref tested, dashboard link |
| 79 | + - Label: `bug`, `automated`, `external-validation` |
| 80 | + |
| 81 | +5. **Schedule**: Weekly (matches current cron). Optionally trigger on argus releases via `repository_dispatch` if cross-repo token is available. |
| 82 | + |
| 83 | +### What argus-test Does NOT Do |
| 84 | +- Does not gate argus PRs (cross-repo dispatch is fragile, static ref limitation) |
| 85 | +- Does not duplicate in-repo tests (unit tests, E2E tests already comprehensive) |
| 86 | +- Does not require maintenance for every argus change (tests use published interfaces) |
0 commit comments