Skip to content

Commit 0e9cb3a

Browse files
committed
docs(sdk): add testing strategy with ref lifecycle and argus-test plan
Documents the three-stage ref lifecycle (PR → main → release) and the division between in-repo testing (PR gating) and external validation (argus-test for release confidence). In-repo: 403+ pytest tests, E2E actions, CLI validation, container builds, audit trail — all on every PR. No external deps. argus-test: scheduled weekly against latest release, opens issues on argus when failures detected, validates consumer perspective. Referenced from issue #29 comment.
1 parent 0ae92c8 commit 0e9cb3a

1 file changed

Lines changed: 86 additions & 0 deletions

File tree

docs/developer/testing-strategy.md

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Argus Testing Strategy
2+
3+
## Ref Lifecycle
4+
5+
Version references flow through three stages, each automatically managed:
6+
7+
### PR Branch (feature development)
8+
- Composite actions: `./` (local path — tests the PR's code)
9+
- Reusable workflows: `./` (local path via workflow_call)
10+
- `test-actions.yml` E2E: uses local checkout, validates the PR branch
11+
- `build-containers.yml`: builds images from the PR, scans with argus CLI from the PR
12+
- No version tags involved — everything is local
13+
14+
### Main Branch (after merge)
15+
- Composite actions: still `./` for internal CI
16+
- Reusable workflows: callers outside the repo use `@main`
17+
- `security-scan.yml`: runs argus scan against main (scheduled weekly + on push)
18+
- All tests re-run on push to main as final validation
19+
20+
### Release Tag (cut by release-it)
21+
- `release-it` bumps all version refs across the repo:
22+
- `version.yaml` (source of truth)
23+
- `package.json` `"version":`
24+
- `argus/__init__.py` `__version__ =`
25+
- `argus/containers.py` GHCR image tags
26+
- All `@X.Y.Z` refs in actions, workflows, examples, docs
27+
- Container images published to GHCR with version tag + `latest`
28+
- GitHub Release created with changelog
29+
30+
## In-Repo Testing (PR Pipeline)
31+
32+
All validation happens within the argus repo. No external repos required for PR gating.
33+
34+
### Test Matrix
35+
36+
| Workflow | What It Tests | Trigger |
37+
|----------|--------------|---------|
38+
| `test-unit.yml` | pytest suite (403+ tests), Python 3.11/3.12/3.13, version ref coverage | PR, push to main |
39+
| `test-actions.yml` | Composite action E2E with real scanners against test fixtures | PR (path-filtered), push to main |
40+
| `build-containers.yml` | Build images → scan with Trivy+Grype → test argus CLI → validate audit trail | PR (path-filtered) |
41+
| `test-examples-functional.yml` | Example workflow YAML validation | PR (path-filtered) |
42+
| `test-lint-workflows.yml` | actionlint on all workflow files | PR, push to main |
43+
| `security-scan.yml` | Dogfood: argus scans argus | push to main, weekly schedule |
44+
45+
### Coverage Guarantees
46+
47+
- **Unit tests**: 80%+ coverage enforced via pytest-cov
48+
- **Composite actions**: Every scanner action has E2E test in test-actions.yml
49+
- **Argus CLI**: build-containers.yml tests --list, scan, output validation, SARIF structure, audit trail
50+
- **Container images**: Built, scanned with Trivy+Grype, results posted as PR comment
51+
- **Version refs**: check-version-refs.py runs in CI, fails if any ref would be stale after release
52+
53+
## External Validation: argus-test
54+
55+
The `huntridge-labs/argus-test` repo serves as an **external consumer validation** that runs on a schedule against the latest published release. It does NOT gate PRs.
56+
57+
### Purpose
58+
- Validates that published releases work from a consumer's perspective
59+
- Catches regressions that in-repo tests miss (real-world usage patterns)
60+
- Generates a dashboard with historical pass/fail rates
61+
- Alerts the team when the latest release breaks
62+
63+
### What argus-test Needs (TODO)
64+
65+
1. **Update to 1.0.0 interfaces**: The existing test workflows call `container-scan.yml`, `infrastructure-scan.yml`, etc. These were restored with the same interfaces but now use argus CLI internally. Tests should still pass.
66+
67+
2. **Add argus CLI tests**: New test category that validates:
68+
- `argus scan --list` returns all scanners
69+
- `argus scan bandit --path <test-fixtures>` produces expected findings
70+
- `argus scan container --image nginx:1.19.0` detects known CVEs
71+
- `argus collect` merges multi-scanner results
72+
- `argus-audit.json` is produced with valid structure
73+
74+
3. **Add SDK unit tests**: Run `pytest argus/tests/` against the published ref to ensure the SDK works when consumed via checkout (before PyPI is available).
75+
76+
4. **Alerting**: When the scheduled run fails:
77+
- Open an issue on `huntridge-labs/argus` with the failure details
78+
- Include: which tests failed, the argus ref tested, dashboard link
79+
- Label: `bug`, `automated`, `external-validation`
80+
81+
5. **Schedule**: Weekly (matches current cron). Optionally trigger on argus releases via `repository_dispatch` if cross-repo token is available.
82+
83+
### What argus-test Does NOT Do
84+
- Does not gate argus PRs (cross-repo dispatch is fragile, static ref limitation)
85+
- Does not duplicate in-repo tests (unit tests, E2E tests already comprehensive)
86+
- Does not require maintenance for every argus change (tests use published interfaces)

0 commit comments

Comments
 (0)