Skip to content

Commit 1b212f1

Browse files
authored
feat(skill): github-repo-setup agent-led compliance validation (#26)
* feat(setup): add github-repo-setup validation skill * fix(github-repo-setup): resolve PR feedback on API checks, test coverage, and temp dir * fix(github-repo-setup): address copilot pr feedback on maturity check script * fix(skill): address review findings in github-repo-setup - Remove action-test-and-analyse special-casing from code quality, security supply chain, and knip checks; those are one tool among many and should not receive implicit scoring credit - Fix double-scored rolling update check in check_deployment(); remove the simpler first block, keep the thorough dynamic manifest discovery - Renumber check_ci_cd() comments sequentially (5-10, were skipping and repeating 5 and 6) - Rename Java ESLint-compensation log message to accurately describe it as an auto-grant for N/A tooling rather than a quality signal - Collapse knip security check from elif chain into a single || condition - Expand supply chain audit to also detect audit steps in workflows - Clarify SKILL.md: dependency age and vulnerability SLAs are manual review items; the script cannot verify them automatically - Add trailing newline to SKILL.md * refactor: remove bash script; skill is agent-led - Delete skills/github-repo-setup/scripts/maturity-check.sh - Skill design specifies agent-led execution via SKILL.md workflow - SKILL.md + REPORT_TEMPLATE.md provide complete specification - Agents (Copilot, Explore) have all necessary tools (grep, file read, git commands) - Script approach duplicated logic and had maintenance issues - Aligns with decision to convert to agent-led compliance auditing * docs(skill): convert github-repo-setup to agent-led with renovate tracing - Replace script-based workflow with agent-led approach - Agents directly inspect files, run CLI commands, and generate reports - Add Renovate Preset Tracing methodology (5-step process) - Clarify that auditors must trace inherited presets for effective config - Remove references to maturity-check.sh - Add Output Format section referencing REPORT_TEMPLATE.md - Update description to reflect report generation approach * docs(template): add report template with renovate inheritance analysis - Add REPORT_TEMPLATE.md for maturity assessment output format - Include Renovate Configuration Inheritance Analysis section - Provides structure for agents to document effective renovate config - Supports auditors in understanding preset inheritance chains - References ASVS-aligned compliance checklist * docs(skill): add examples, edge cases, and references sections * fix: address copilot PR feedback on skill and template - Remove reference to non-existent TEAM_CHECKLIST.md file - Update template path to use relative markdown link - Replace hardcoded status check names with reference to aggregator pattern in github-actions skill - Fix capitalization: BCGov → BC Gov * docs(skill): add explicit gh CLI guidance for branch protection verification - Require use of `gh repo view --json branchProtectionRules` to inspect rulesets - If gh CLI unavailable, agent must stop and ask user for manual verification - Prevents false negatives when API access is unavailable - Add edge case warning about gh CLI dependency * fix: use correct gh rulesets API instead of deprecated branchProtectionRules - GitHub moved to rulesets API; branchProtectionRules is deprecated - Update workflow to query rulesets with rule types: PULL_REQUEST, REQUIRED_STATUS_CHECKS, NON_FAST_FORWARD - Verify enforcement is ACTIVE on main branch - Add edge case guidance for rulesets API vs deprecated API - Update references to gh api documentation * refactor(skill): implement high-priority fixes from repo owner review - Create references/REFERENCE.md with comprehensive scoring rubric, 9 dimension detailed rules, SLA workflow, image promotion pattern, EPSS guidance - Reconcile Rules & template with all 9 dimensions (Dims 1, 4, 6, 7, 8 now fully defined) - Add GitHub API requirement for Dims 1–2; introduce Unverified state (distinct from Not Met) - Expand Workflow steps to map all 9 dimensions for comprehensive coverage - Cross-link openshift-deployment SKILL for remediation guidance - Update REPORT_TEMPLATE.md with Scoring Formula section and Unverified status markers - Add trigger phrases to description for routing (audit, compliance, scorecard, maturity) - Clarify Renovate preset version as illustrative Addresses: - HIGH: Rules/template out of sync (missing Dims 4, 6, 7, 8, 1) - HIGH: No scoring rubric (now explicit, reproducible) - HIGH: Server-side settings have no data source (now explicit GitHub API requirement + Unverified state) - MEDIUM: OpenShift overlap (now cross-referenced) - MEDIUM: Workflow doesn't cover all dimensions (now maps to all 9) - MEDIUM: Heavy detail in manifest (now in references/REFERENCE.md) * refactor(github-repo-setup): simplify skill to agent-led format and consolidate templates * refactor(github-repo-setup): add stable tags vs digests edge case rule * refactor(github-repo-setup): clarify container image scanning is optional and non-blocking * refactor(github-repo-setup): address Opus review comments on deterministic scoring and legends
1 parent b885a1c commit 1b212f1

2 files changed

Lines changed: 555 additions & 0 deletions

File tree

skills/github-repo-setup/SKILL.md

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
---
2+
name: github-repo-setup
3+
description: 'Assess GitHub repository maturity against BC Gov DevOps and security standards. Generates a MATURITY_REPORT.md scorecard covering branch protection rulesets, TypeScript strictness, Renovate preset tracing, vulnerability SLAs, CI/CD quality gates, and OpenShift container security. Use when onboarding repos, running compliance reviews, or evaluating vendor conformance.'
4+
owner: bcgov
5+
tags: [github-repo-setup, devops, security, bcgov, audit, compliance]
6+
---
7+
8+
# GitHub Repository Setup Validation
9+
10+
Evaluate repository compliance against BC Gov DevOps and security standards by inspecting config files, querying GitHub APIs, and generating a structured maturity report.
11+
12+
## Use When
13+
- Onboarding a new repository to a BC Gov GitHub organization.
14+
- Preparing for a security audit, compliance review, or sprint planning.
15+
- Evaluating a repository before transitioning it to maintenance mode.
16+
- Validating vendor compliance with BC Gov digital standards.
17+
18+
## Don't Use When
19+
- Auditing personal, experimental, or toy repositories, or repository templates/scaffolding not containing production content. (BC Gov documentation-only or basic scripting repositories ARE in scope; assess them with inapplicable dimensions marked N/A).
20+
- Performing actual resource provisioning, template setup, or automated repository scaffolding (this skill is purely an assessment and compliance validation auditor).
21+
22+
## Workflow
23+
24+
1. **Analyze Project Structure** — Walk the repo root and identify: `package.json`, `tsconfig.json`, `renovate.json` or `dependabot.yml`, `.github/workflows/`, OpenShift/Kubernetes manifests, `SECURITY.md`, `.github/ISSUE_TEMPLATE/`.
25+
26+
2. **Inspect Server-side Config** (Dimensions 1–2) — Use `gh api graphql` to query repository settings and branch rulesets. If `gh` CLI is unavailable, mark findings as **Unverified** (distinct from Not Met). See [REFERENCE.md query example](./references/REFERENCE.md#dimension-2-branch-protection-rulesets).
27+
28+
3. **Inspect Code and File-based Config** (Dimensions 3–5, 8–9):
29+
- Read `tsconfig.json` for `strict: true`; grep for `@ts-ignore`, `@ts-nocheck`, `eslint-disable`.
30+
- Read `renovate.json` — trace preset inheritance through `extends` entries to determine effective automerge, schedule, and minimumReleaseAge settings.
31+
- Read `.github/workflows/` for test/coverage/scan gates.
32+
- Read deployment manifests for `runAsNonRoot`, `readOnlyRootFilesystem`, probes.
33+
34+
4. **Assess Processes and Docs** (Dimensions 6–7) — Read `SECURITY.md` and issue templates for documented triage workflow and SLAs. Read workflows for image promotion patterns.
35+
36+
5. **Draft and Write Report** — Score each of the 9 dimensions using the formula: `Compliance Score (%) = (Sum of Weighted Dimension Scores / Total Scored Dimensions) × 100` (where Met = 1.0, Partial = 0.5, Not Met = 0.0, and Unverified/N/A are excluded from both numerator and denominator). Calculate compliance percentage, map to Maturity Level 1–5, and generate `MATURITY_REPORT.md` in the audited repo root using the [report template](./references/REFERENCE.md#report-template).
37+
38+
## Rules
39+
40+
All dimension definitions and scoring details are in [REFERENCE.md](./references/REFERENCE.md#2-nine-compliance-dimensions--detailed-rules).
41+
42+
### Nine Compliance Dimensions
43+
44+
- **Dim 1 – Repo Settings**: Squash merging enforced, auto-cleanup, suggest updates. **Requires GitHub API.**
45+
- **Dim 2 – Branch Rulesets**: PR requirement, approvals, linear history, status checks. **Requires `gh api graphql`.**
46+
- **Dim 3 – Code Hygiene**: TS strict mode, linting enforced, no diagnostic escapes, 80%+ test coverage.
47+
- **Dim 4 – Secrets**: Token/password separation per environment, 32+ char passwords.
48+
- **Dim 5 – Dependency Updates**: Renovate/Dependabot configured, BC Gov preset preferred, 7-day minimum release age, automerge enabled. **Trace preset inheritance.**
49+
- **Dim 6 – Vulnerability SLAs**: Documented triage workflow, CISA KEV monitoring, FIRST EPSS scoring. Critical 24h / High 1w / Medium 2w / Low next scheduled release.
50+
- **Dim 7 – CI/CD and Deployments**: PR preview envs, image promotion (no rebuilds), SHA-based image references, deployment health gating.
51+
- **Dim 8 – Quality Gates**: TS/lint/test/coverage/scan failures block merge.
52+
- **Dim 9 – OpenShift Security**: Pod security contexts (runAsNonRoot, readOnlyFS), capabilities dropped, seccomp, probes. For remediation, defer to [openshift-deployment SKILL](../openshift-deployment/SKILL.md).
53+
54+
### Critical Rules
55+
56+
- **No-Exemption Policy**: All security vulnerabilities must be remediated. No exceptions for "trusted environments" or "internal access."
57+
- **Unverified vs Not Met**: When a data source (GitHub API, `gh` CLI) is unavailable, mark as **Unverified** — not "Not Met." Do not fabricate findings.
58+
- **Renovate Preset Tracing**: Trace all `extends` entries, fetch referenced preset repos, merge with local overrides, report effective settings.
59+
- **Cross-Reference**: For OpenShift manifest remediation, defer to [openshift-deployment SKILL](../openshift-deployment/SKILL.md).
60+
61+
## Output Format
62+
63+
Generate `MATURITY_REPORT.md` in the audited repo root using the [report template in REFERENCE.md](./references/REFERENCE.md#report-template).
64+
65+
**Report includes:** Executive Summary (score, level, date), 9-dimension breakdown table, detailed checklist with legend markers (`[x]` Met, `[~]` Partial, `[ ]` Not Met, `[?]` Unverified, `[-]` N/A), scoring formula, and prioritized remediation (Tier 1 blocking, Tier 2 recommended, Tier 3 optional).
66+
67+
## Examples
68+
69+
### Example 1: Production Microservice
70+
**Prompt**: "Assess bcgov/my-api-service for compliance"
71+
**Process**: Walk repo → find TypeScript + Renovate + OpenShift manifests → verify `strict: true` → trace Renovate preset inheritance → check pod security contexts → score 85% (gaps: missing PDB, coverage at 78%) → generate report.
72+
**Output**: `MATURITY_REPORT.md` with Level 4 (Managed), 2 Tier-2 remediation items.
73+
74+
### Example 2: New Repository Onboarding
75+
**Prompt**: "Is this repo ready for BC Gov production?"
76+
**Process**: Walk repo → find React app, no `renovate.json`, no branch ruleset → score 55% → flag Tier-1 blockers.
77+
**Output**: `MATURITY_REPORT.md` with Level 3 (Defined), 4 Tier-1 blocking items.
78+
79+
## Edge Cases
80+
81+
- **Monorepos**: Inspect root `tsconfig.json` and workspace-level configs. Flag inconsistencies across packages.
82+
- **Non-TypeScript Repos**: Skip TS-specific rules. Assess language-appropriate linting (ESLint for JS, Pylint for Python).
83+
- **GitHub API Unavailable**: Mark Dims 1–2 as **Unverified**. Example: "Unverified (GitHub API access required). Recommend: `gh auth login` and re-run."
84+
- **Renovate Config Without Extends**: Note default Renovate behavior applies. Verify manual config is comprehensive.
85+
- **Legacy Branch Names** (`master`, `develop`): Audit still applies. Flag as low-priority gap.
86+
- **Exempt Repos**: Documentation-only or archived repos — mark dimensions N/A with documented reason.
87+
- **Reproducibility**: Two runs on the same repo must produce identical scores. Use explicit rubric; mark Unverified rather than guessing.
88+
- **Stable Tags vs. Cryptographic Digests**: Using unique stable tags (e.g. Git commit SHAs, PR numbers) solves basic config drift but lacks registry-level immutability. Score the SHA-based image references check as **Partial** (not Met). Only true container image digests (`@sha256:...`) qualify for **Met**.
89+
90+
91+
## References
92+
93+
- [REFERENCE.md](./references/REFERENCE.md) — Scoring rubric, 9 dimension definitions, SLA workflow, image promotion pattern, report template, GraphQL query examples.
94+
- [github-actions SKILL](../github-actions/SKILL.md) — Status check aggregator pattern and CI/CD best practices.
95+
- [openshift-deployment SKILL](../openshift-deployment/SKILL.md) — OpenShift pod security contexts and manifest authoring.
96+
- [BC Gov Renovate Config](https://github.com/bcgov/renovate-config) — Inherited preset configurations.
97+
- [GitHub CLI Reference](https://cli.github.com/manual/gh_api)`gh api` for GraphQL queries.
98+
- [CISA KEV](https://www.cisa.gov/known-exploited-vulnerabilities) — Actively exploited CVE tracking.
99+
- [FIRST EPSS](https://www.first.org/epss/) — Exploit Prediction Scoring System.

0 commit comments

Comments
 (0)