feat: validate config files against published JSON Schema at runtime by lpcox · Pull Request #2384 · github/gh-aw-firewall

lpcox · 2026-05-01T23:49:42Z

Summary

Replace the hand-written validateAwfFileConfig() with ajv-based validation against the published awf-config.schema.json. This makes the JSON Schema the single source of truth for:

Runtime validation inside awf (this PR)
External consumers like the gh-aw compiler (schema is downloadable as a release asset and from docs/awf-config.schema.json)
IDE autocomplete via the $schema field

Changes

File	Change
`package.json`	Move `ajv` from devDependencies → dependencies
`src/schema-validator.ts`	New module: compiles schema once, formats ajv errors into human-readable messages
`src/awf-config-schema.json`	Bundleable copy of the schema (imported at compile time for esbuild compat)
`scripts/generate-schema.mjs`	Also writes to `src/` alongside `docs/`
`src/config-file.ts`	Delegates to schema-validator (removes ~170 lines of hand-written validation)
`src/schema.test.ts`	Adds sync-check test (src/ and docs/ schemas stay aligned)
`src/config-file.test.ts`	One assertion updated for improved enum format

Design decisions

Verbose mode (ajv({ verbose: true })) to access parentSchema for rich error messages
Error formatting layer maps ajv errors into stable messages matching previous UX
Module-level singleton — schema compiled once, reused across all validations
Bundle-safe — JSON schema is a TypeScript import (no runtime fs.readFileSync)

How gh-aw compiler uses this

The compiler can fetch the schema from:

Release asset: https://github.com/github/gh-aw-firewall/releases/latest/download/awf-config.schema.json
Raw: https://raw.githubusercontent.com/github/gh-aw-firewall/main/docs/awf-config.schema.json

and validate configs client-side before passing them to awf.

Closes #2374

Replace the hand-written validateAwfFileConfig() implementation with ajv-based validation against the published awf-config.schema.json. This makes the JSON Schema the single source of truth for: 1. Runtime validation inside awf (this change) 2. External consumers like the gh-aw compiler (schema is downloadable as a release asset and from docs/awf-config.schema.json) 3. IDE autocomplete via $schema field Changes: - Move ajv from devDependencies to dependencies (runtime use) - Add src/schema-validator.ts: compiles schema once, formats ajv errors into human-readable messages matching the previous validator style - Add src/awf-config-schema.json: bundleable copy of the schema (imported at compile time, works with esbuild single-file bundle) - Update generate-schema.mjs to write both docs/ and src/ copies - Add sync test ensuring src/ and docs/ schemas stay aligned - Update one test assertion for new enum formatting The error messages are backward-compatible with the previous hand-written validator (all 113 config/schema tests pass unchanged except one enum format improvement: '"5m" or "1h"' → 'one of: 5m, 1h'). Closes #2374 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-01T23:50:41Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	85.90%	85.82%	📉 -0.08%
Statements	85.88%	85.70%	📉 -0.18%
Functions	88.00%	88.18%	📈 +0.18%
Branches	80.06%	78.55%	📉 -1.51%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)
`src/config-file.ts`	97.0% → 100.0% (+3.02%)	96.6% → 98.0% (+1.42%)

✨ New Files (1 files)

src/schema-validator.ts: 96.4% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

lpcox · 2026-05-01T23:50:46Z

@copilot the schema needs to be versioned

Copilot

Pull request overview

Introduces AJV-based runtime validation for AWF config files using the published awf-config.schema.json, aiming to make the JSON Schema the single source of truth for validation across AWF, external tooling, and IDEs.

Changes:

Add src/schema-validator.ts to compile and run AJV schema validation with human-readable error formatting.
Replace hand-written config validation in src/config-file.ts with schema-based validation.
Generate and add a bundleable schema copy in src/awf-config-schema.json, plus a test to keep it in sync with docs/.

Show a summary per file

File	Description
package.json	Moves `ajv` into runtime dependencies.
src/schema-validator.ts	New AJV-based validator + error formatting/deduplication.
src/config-file.ts	Delegates config validation to the schema validator.
src/awf-config-schema.json	Adds schema copy intended for runtime/bundling.
scripts/generate-schema.mjs	Writes schema output to `src/` as well as `docs/`.
src/schema.test.ts	Adds sync-check ensuring `src/` schema matches `docs/` schema (minus `$id`).
src/config-file.test.ts	Updates expected enum error message format.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 7/7 changed files
Comments generated: 2

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-05-01T23:55:40Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	85.90%	85.80%	📉 -0.10%
Statements	85.88%	85.68%	📉 -0.20%
Functions	88.00%	88.18%	📈 +0.18%
Branches	80.06%	78.51%	📉 -1.55%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)
`src/config-file.ts`	97.0% → 100.0% (+3.02%)	96.6% → 98.0% (+1.42%)

✨ New Files (1 files)

src/schema-validator.ts: 96.2% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-05-01T23:55:42Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	85.90%	85.80%	📉 -0.10%
Statements	85.88%	85.68%	📉 -0.20%
Functions	88.00%	88.21%	📈 +0.21%
Branches	80.06%	78.49%	📉 -1.57%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)
`src/config-file.ts`	97.0% → 100.0% (+3.02%)	96.6% → 98.0% (+1.42%)

✨ New Files (1 files)

src/schema-validator.ts: 94.8% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b9f23f04-41af-4ebf-ba7f-52aa520cad1d

github-actions · 2026-05-01T23:59:01Z

Smoke Test Result:

GitHub MCP: ✅
GitHub.com Connectivity: ✅
File Write/Read Test: ✅

Overall Status: PASS
@lpcox

💎 Faceted by Smoke Gemini

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b9f23f04-41af-4ebf-ba7f-52aa520cad1d

Copilot · 2026-05-02T00:03:53Z

@copilot the schema needs to be versioned

Done in bafaae2. Following the same pattern as schemas/audit.v1.schema.json:

Added "version": "1" to the schema body
Created docs/awf-config.v1.schema.json as a stable versioned file with a v1-suffixed $id URL
docs/awf-config.schema.json remains as the "latest" alias
Release workflow now publishes both awf-config.v1.schema.json and awf-config.schema.json as release assets
Registered the version keyword with Ajv so strict mode doesn't reject it

lpcox · 2026-05-02T00:05:33Z

@copilot all schema versions should be tied to a release tag. is there a way to do this in release.md?

…asing.md Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/72cdaa37-37d8-4564-937b-e5f2ee8eafc1 Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>

Copilot · 2026-05-02T00:07:12Z

@copilot all schema versions should be tied to a release tag. is there a way to do this in release.md?

Done in 4b4145e. Added a "JSON Schema versioning" section to docs/releasing.md that explains:

Each release embeds the tag in the schema's $id URL (e.g. .../releases/download/v0.23.1/awf-config.v1.schema.json), creating a stable, pinnable reference per release
A table of URL options: pinned-to-tag vs always-latest-from-main
Guidance on when to bump the schema version number (breaking changes only) and how to introduce a v2 file when that happens
Schema files added to the release verification checklist and the GitHub Release Assets listing

github-actions · 2026-05-02T00:08:37Z

Documentation Preview

Documentation build failed for this PR. View logs.

Built from commit ea01710

github-actions · 2026-05-02T00:08:58Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	85.90%	85.80%	📉 -0.10%
Statements	85.88%	85.68%	📉 -0.20%
Functions	88.00%	88.21%	📈 +0.21%
Branches	80.06%	78.49%	📉 -1.57%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)
`src/config-file.ts`	97.0% → 100.0% (+3.02%)	96.6% → 98.0% (+1.42%)

✨ New Files (1 files)

src/schema-validator.ts: 94.9% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

- Add 18 unit tests covering all error formatting paths - Remove dead code: hasMinimumOne() and determineArticle() functions (all integer fields have minimum:1; no vowel-starting types reach the article logic after prior case handling) - Simplify isArrayOfStringsField() to a single boolean expression - Branch coverage for schema-validator.ts: 97.77% Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-02T00:27:24Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	85.90%	85.83%	📉 -0.07%
Statements	85.88%	85.71%	📉 -0.17%
Functions	88.00%	88.15%	📈 +0.15%
Branches	80.06%	78.67%	📉 -1.39%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)
`src/config-file.ts`	97.0% → 100.0% (+3.02%)	96.6% → 98.0% (+1.42%)

✨ New Files (1 files)

src/schema-validator.ts: 98.0% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

- Conditionally switch to azure.archive.ubuntu.com only if DNS resolves it - Falls back to default archive.ubuntu.com when BuildKit DNS can't reach the Azure mirror (common in Docker-in-Docker scenarios) - Improve apt_update_retry() to detect silent failures (apt returns 0 with 'Failed to fetch' warnings) and fall back to archive.ubuntu.com - Fixes persistent CI failures where BuildKit couldn't resolve azure.archive.ubuntu.com during container image builds Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-02T00:42:28Z

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric	Base	PR	Delta
Lines	85.90%	85.83%	📉 -0.07%
Statements	85.88%	85.71%	📉 -0.17%
Functions	88.00%	88.15%	📈 +0.15%
Branches	80.06%	78.67%	📉 -1.39%

📁 Per-file Coverage Changes (2 files)

File	Lines (Before → After)	Statements (Before → After)
`src/docker-manager.ts`	87.4% → 87.7% (+0.29%)	87.0% → 87.3% (+0.27%)
`src/config-file.ts`	97.0% → 100.0% (+3.02%)	96.6% → 98.0% (+1.42%)

✨ New Files (1 files)

src/schema-validator.ts: 98.0% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-05-02T00:45:10Z

🤖 Smoke Test Results

Test	Result
GitHub MCP connectivity	✅ PASS
GitHub.com HTTP	❌ Pre-step data unavailable (template vars unexpanded)
File write/read	❌ Pre-step data unavailable (template vars unexpanded)

PR: feat: validate config files against published JSON Schema at runtime
Author: @lpcox | Reviewers: @Mossaka

Overall: FAIL — smoke-data outputs were not populated by the pre-agent step.

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-05-02T00:45:10Z

Smoke Test Results:
✅ GitHub MCP: #2380, #2375
✅ Playwright: github.com title verified
✅ File Writing: test file created
✅ Bash Tool: file verified
Status: PASS

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-05-02T00:46:45Z

Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	`Python 3.12.13`	`Python 3.12.3`	❌
Node.js	`v24.14.1`	`v20.20.2`	❌
Go	`go1.22.12`	`go1.22.12`	✅

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

github-actions · 2026-05-02T00:46:53Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	passed	✅ PASS
Go	env	✅	passed	✅ PASS
Go	uuid	✅	passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	passed	✅ PASS
Node.js	execa	✅	passed	✅ PASS
Node.js	p-limit	✅	passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Note: Java initially failed with a local Maven repository permission error (/home/runner/.m2 was owned by root). Worked around by using -Dmaven.repo.local=/tmp/gh-aw/agent/m2repo.

Generated by Build Test Suite for issue #2384 · ● 561.6K · ◷

github-actions · 2026-05-02T00:49:54Z

Smoke Test Results

Check	Result
Redis PING	❌ timeout/unreachable
PostgreSQL pg_isready	❌ no response
PostgreSQL SELECT 1	❌ no response

Overall: FAIL — host.docker.internal is not reachable from this runner environment. Service containers may not be running or networking is misconfigured.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-05-02T01:14:25Z

Smoke Test: Copilot BYOK (Offline) Mode

Test	Result
GitHub MCP (list PRs)	✅ PR #2384 fetched
GitHub.com connectivity	⚠️ Pre-step data not substituted
File write/read	⚠️ Pre-step data not substituted
BYOK inference (this response)	✅

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

Author: @lpcox | Reviewer: @Mossaka

⚠️ Status: PARTIAL — Tests 2 & 3 skipped due to unsubstituted workflow template variables ($\{\{ steps.smoke-data.outputs.* }}). BYOK inference and GitHub MCP confirmed working.

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-05-02T01:15:16Z

feat: publish versioned schemas for JSONL audit/telemetry artifacts
feat: publish versioned JSON Schema for AWF config file
✅ GitHub PR review
❌ safeinputs-gh query
✅ Playwright
❌ Tavily
✅ File write/readback
✅ Discussion comment
✅ Build
Overall: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

Copilot AI review requested due to automatic review settings May 1, 2026 23:49

lpcox requested a review from Mossaka as a code owner May 1, 2026 23:49

Copilot started work on behalf of lpcox May 1, 2026 23:50 View session

This was referenced May 1, 2026

[aw] Smoke Copilot failed #2385

Open

[aw] Smoke Copilot BYOK failed #2386

Open

[aw] Smoke Chroot failed #2387

Open

[aw] Smoke Codex failed #2388

Open

[aw] Smoke Claude failed #2389

Open

[aw] Build Test Suite failed #2390

Open

Copilot AI reviewed May 1, 2026

View reviewed changes

Comment thread src/schema-validator.ts Outdated

Comment thread src/schema-validator.ts Outdated

lpcox and others added 2 commits May 1, 2026 16:54

Potential fix for pull request finding

bde7732

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

be61364

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

feat: version the awf-config JSON Schema as v1

d9bd558

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b9f23f04-41af-4ebf-ba7f-52aa520cad1d

github-actions Bot added the smoke-gemini label May 1, 2026

Copilot AI added 2 commits May 2, 2026 00:00

feat: merge upstream schema-validator changes with schema versioning

8b1638d

feat: version the awf-config JSON Schema as v1

bafaae2

Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/b9f23f04-41af-4ebf-ba7f-52aa520cad1d

Copilot finished work on behalf of lpcox May 2, 2026 00:04

Copilot started work on behalf of lpcox May 2, 2026 00:05 View session

docs: document JSON Schema versioning and release tag pinning in rele…

4b4145e

…asing.md Agent-Logs-Url: https://github.com/github/gh-aw-firewall/sessions/72cdaa37-37d8-4564-937b-e5f2ee8eafc1 Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>

Copilot finished work on behalf of lpcox May 2, 2026 00:07

github-actions Bot mentioned this pull request May 2, 2026

[aw] Smoke Services failed #2391

Open

This comment has been minimized.

Sign in to view

github-actions Bot added the smoke-claude label May 2, 2026

This comment has been minimized.

Sign in to view

github-actions Bot added the build-test label May 2, 2026

github-actions Bot mentioned this pull request May 2, 2026

[aw] No-Op Runs #2301

Open

This comment has been minimized.

Sign in to view

lpcox merged commit 08cd698 into main May 2, 2026
66 of 70 checks passed

lpcox deleted the schema-based-config-validation branch May 2, 2026 02:13

Conversation

lpcox commented May 1, 2026

Summary

Changes

Design decisions

How gh-aw compiler uses this

Uh oh!

github-actions Bot commented May 1, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

lpcox commented May 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 1, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented May 1, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented May 1, 2026

Uh oh!

Copilot AI commented May 2, 2026

Uh oh!

lpcox commented May 2, 2026

Uh oh!

Copilot AI commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation Preview

Uh oh!

github-actions Bot commented May 2, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented May 2, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions Bot commented May 2, 2026

⚠️ Coverage Regression Detected

Overall Coverage

Uh oh!

github-actions Bot commented May 2, 2026

🤖 Smoke Test Results

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented May 2, 2026

Chroot Version Comparison Results

Uh oh!

github-actions Bot commented May 2, 2026

🏗️ Build Test Suite Results

Uh oh!

This comment has been minimized.

github-actions Bot commented May 2, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented May 2, 2026

Smoke Test: Copilot BYOK (Offline) Mode

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

github-actions Bot commented May 2, 2026 •

edited

Loading