Skip to content

Commit 0a56a9e

Browse files
mgiffordmgifford
authored
Merge pull request #1 from mgifford/002-daily-dap-quality-benchmarking-WP05
feat: daily DAP scan workflow and orchestration
2 parents 9d3d2a1 + e1e1e73 commit 0a56a9e

49 files changed

Lines changed: 2962 additions & 109 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/daily-scan.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
name: Daily DAP Scan
2+
3+
on:
4+
schedule:
5+
- cron: '17 9 * * *'
6+
workflow_dispatch:
7+
inputs:
8+
run_date:
9+
description: 'Optional run date (YYYY-MM-DD)'
10+
required: false
11+
type: string
12+
url_limit:
13+
description: 'Optional URL limit override'
14+
required: false
15+
type: string
16+
traffic_window:
17+
description: 'Traffic window mode (daily|rolling_7d|rolling_30d)'
18+
required: false
19+
default: 'daily'
20+
type: choice
21+
options:
22+
- daily
23+
- rolling_7d
24+
- rolling_30d
25+
dry_run:
26+
description: 'Run in dry-run mode only'
27+
required: false
28+
type: boolean
29+
default: false
30+
31+
permissions:
32+
contents: write
33+
34+
concurrency:
35+
group: daily-dap-scan
36+
cancel-in-progress: false
37+
38+
jobs:
39+
daily-scan:
40+
runs-on: ubuntu-latest
41+
env:
42+
NODE_ENV: production
43+
RUN_DATE: ${{ inputs.run_date }}
44+
URL_LIMIT: ${{ inputs.url_limit }}
45+
TRAFFIC_WINDOW: ${{ inputs.traffic_window || 'daily' }}
46+
DRY_RUN: ${{ inputs.dry_run || false }}
47+
steps:
48+
- name: Checkout
49+
uses: actions/checkout@v4
50+
with:
51+
fetch-depth: 0
52+
53+
- name: Setup Node
54+
uses: actions/setup-node@v4
55+
with:
56+
node-version: '20'
57+
cache: 'npm'
58+
59+
- name: Install dependencies
60+
run: npm ci
61+
62+
- name: Run tests
63+
run: npm test
64+
65+
- name: Run pipeline
66+
id: pipeline
67+
shell: bash
68+
run: |
69+
set +e
70+
ARGS=(--scan-mode mock)
71+
if [[ -n "$RUN_DATE" ]]; then ARGS+=(--date "$RUN_DATE"); fi
72+
if [[ -n "$URL_LIMIT" ]]; then ARGS+=(--limit "$URL_LIMIT"); fi
73+
if [[ -n "$TRAFFIC_WINDOW" ]]; then ARGS+=(--traffic-window "$TRAFFIC_WINDOW"); fi
74+
if [[ "$DRY_RUN" == "true" ]]; then ARGS+=(--dry-run); fi
75+
node src/cli/run-daily-scan.js "${ARGS[@]}"
76+
EXIT_CODE=$?
77+
echo "exit_code=$EXIT_CODE" >> "$GITHUB_OUTPUT"
78+
exit 0
79+
80+
- name: Upload diagnostics artifact
81+
if: always()
82+
uses: actions/upload-artifact@v4
83+
with:
84+
name: daily-dap-run-artifacts
85+
path: |
86+
artifacts/
87+
docs/reports/daily/
88+
if-no-files-found: warn
89+
90+
- name: Commit report snapshots
91+
if: always() && steps.pipeline.outputs.exit_code == '0' && env.DRY_RUN != 'true'
92+
shell: bash
93+
run: |
94+
if git diff --quiet -- docs/reports; then
95+
echo "No report snapshot changes to commit."
96+
exit 0
97+
fi
98+
99+
git config user.name "github-actions[bot]"
100+
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
101+
git add docs/reports
102+
git commit -m "chore: publish daily DAP report snapshots"
103+
git push
104+
105+
- name: Fail workflow on pipeline error
106+
if: steps.pipeline.outputs.exit_code != '0'
107+
run: |
108+
echo "Pipeline command exited with code ${{ steps.pipeline.outputs.exit_code }}"
109+
exit 1

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,5 @@
1515
.kittify/missions/__pycache__/
1616
__pycache__/
1717
*.pyc
18+
node_modules/
19+
artifacts/

‎README.md‎

Lines changed: 69 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,69 @@
1-
# daily-dap
2-
A daily scan of the Digital Accessibility Program
1+
# Daily DAP Quality Benchmarking
2+
3+
Daily DAP benchmarks the quality and accessibility of the most visited U.S. government pages.
4+
It prioritizes high-traffic pages because regressions on those pages affect the most people seeking public services.
5+
6+
## Why this project exists
7+
8+
Public-facing government websites are critical infrastructure. When heavily used pages have accessibility,
9+
performance, or usability issues, impact is broad and immediate.
10+
11+
This project provides a daily, repeatable quality signal by:
12+
13+
- pulling top pages from DAP traffic data,
14+
- scanning those pages with Lighthouse and ScanGov,
15+
- aggregating quality and impact metrics,
16+
- publishing dated static reports and trend history.
17+
18+
## DAP and related resources
19+
20+
- Digital Analytics Program (DAP): https://digital.gov/guides/dap/
21+
- Analytics.USA.gov overview: https://analytics.usa.gov/
22+
- DAP data endpoint configured in this repo: `src/config/prevalence.yaml`
23+
- ScanGov (accessibility scanner): https://github.com/GSA/scan-gov
24+
- Lighthouse: https://developer.chrome.com/docs/lighthouse/overview/
25+
26+
## Current implementation status
27+
28+
- WP01–WP04 are implemented through report payload generation, static rendering, archive writing, and schema contract tests.
29+
- WP05 will finalize end-to-end CLI orchestration and scheduled CI automation for the full production run.
30+
31+
## Expected end-to-end action (ingest → scan → report)
32+
33+
The intended operator action is:
34+
35+
1. Pull top DAP URLs and page-load counts.
36+
2. Run Lighthouse + ScanGov scans for each selected URL.
37+
3. Aggregate scores and accessibility impact estimates.
38+
4. Generate and publish dated report snapshots under `docs/reports/`.
39+
40+
This workflow is represented by the CLI entrypoint and work package stack:
41+
42+
- Current entrypoint scaffold: `src/cli/run-daily-scan.js`
43+
- Full orchestration completion target: WP05
44+
45+
## Local development commands
46+
47+
- Install dependencies:
48+
- `npm install`
49+
- Run tests:
50+
- `npm test`
51+
- Run current dry-run pipeline preview:
52+
- `npm run dry-run -- --source-file tests/fixtures/dap-sample.json`
53+
54+
## Output locations
55+
56+
- Daily published snapshots: `docs/reports/daily/YYYY-MM-DD/`
57+
- History index: `docs/reports/history.json`
58+
- Top-level dashboard page: `docs/reports/index.html`
59+
60+
## Project structure
61+
62+
- `src/config/` configuration schema + prevalence inputs
63+
- `src/ingest/` DAP source ingestion + normalization
64+
- `src/scanners/` Lighthouse/ScanGov execution + normalization
65+
- `src/aggregation/` metrics, impact, and trends
66+
- `src/publish/` report building, static rendering, archive + manifest
67+
- `tests/unit/` unit tests
68+
- `tests/contract/` schema contract validation
69+
- `kitty-specs/002-daily-dap-quality-benchmarking/` specification and work packages

‎docs/reports/history.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"generated_at": null,
3+
"lookback_days": 30,
4+
"entries": []
5+
}

‎docs/reports/index.html‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
<!doctype html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="utf-8" />
5+
<meta name="viewport" content="width=device-width, initial-scale=1" />
6+
<title>Daily DAP Reports</title>
7+
</head>
8+
<body>
9+
<h1>Daily DAP Quality Reports</h1>
10+
<p>No reports have been published yet.</p>
11+
</body>
12+
</html>

‎docs/reports/operations.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Daily DAP Operations Runbook
2+
3+
## Operational goals
4+
5+
- Keep daily report publication continuous and transparent.
6+
- Preserve artifact diagnostics for every run.
7+
- Support safe reruns and controlled rollback when publication errors occur.
8+
9+
## Routine execution
10+
11+
- Scheduled execution is handled by `.github/workflows/daily-scan.yml`.
12+
- Manual rerun is available via workflow dispatch inputs:
13+
- `run_date`
14+
- `url_limit`
15+
- `traffic_window`
16+
- `dry_run`
17+
18+
## Local operator commands
19+
20+
- Dry run validation:
21+
- `npm run dry-run -- --source-file tests/fixtures/dap-sample.json`
22+
- Full mock orchestration run:
23+
- `node src/cli/run-daily-scan.js --source-file tests/fixtures/dap-sample.json --scan-mode mock`
24+
- Run a specific date:
25+
- `node src/cli/run-daily-scan.js --date 2026-02-21 --scan-mode mock --source-file tests/fixtures/dap-sample.json`
26+
27+
## Failure handling
28+
29+
When a run fails:
30+
31+
- Failure payload is written to `docs/reports/daily/YYYY-MM-DD/report.json`.
32+
- Failure page is written to `docs/reports/daily/YYYY-MM-DD/index.html`.
33+
- Diagnostics summary is written to `artifacts/YYYY-MM-DD/run-summary.json`.
34+
- Workflow uploads the diagnostics bundle as a GitHub Actions artifact.
35+
36+
## Recovery and rollback
37+
38+
1. Inspect latest workflow run logs and uploaded artifact.
39+
2. Confirm whether failure is environmental (transient) or functional (code/data contract).
40+
3. For transient failures, rerun workflow dispatch for the same date.
41+
4. For functional issues, fix on a branch and rerun manually in dry-run mode first.
42+
5. If published snapshot is bad, revert only affected `docs/reports/` paths and re-run publish workflow.
43+
44+
## Safe config changes
45+
46+
- Edit `src/config/prevalence.yaml` for:
47+
- `scan.url_limit`
48+
- `scan.history_lookback_days`
49+
- `scan.traffic_window_mode`
50+
- impact prevalence rates and severity weights
51+
- Validate with:
52+
- `npm test`
53+
- `npm run dry-run -- --source-file tests/fixtures/dap-sample.json`
54+
55+
## Troubleshooting notes
56+
57+
- Scanner execution failures are expected to produce partial reports rather than silent drops.
58+
- Missing page-load counts are retained for diagnostics and excluded from weighted traffic math.
59+
- Avoid empty snapshot commits: workflow already checks for docs changes before committing.

‎kitty-specs/002-daily-dap-quality-benchmarking/tasks/WP01-pipeline-foundation-and-configuration.md‎

Lines changed: 27 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,29 @@
11
---
2-
work_package_id: "WP01"
3-
subtasks:
4-
- "T001"
5-
- "T002"
6-
- "T003"
7-
- "T004"
8-
- "T005"
9-
title: "Pipeline Foundation and Configuration"
10-
phase: "Phase 1 - Foundation"
11-
lane: "planned"
2+
work_package_id: WP01
3+
title: Pipeline Foundation and Configuration
4+
lane: "done"
125
dependencies: []
13-
assignee: ""
14-
agent: ""
15-
shell_pid: ""
16-
review_status: ""
17-
reviewed_by: ""
6+
base_branch: main
7+
base_commit: 9d3d2a1568d59c415559a214ba0862a372698696
8+
created_at: '2026-02-21T20:15:30.629111+00:00'
9+
subtasks:
10+
- T001
11+
- T002
12+
- T003
13+
- T004
14+
- T005
15+
phase: Phase 1 - Foundation
16+
assignee: ''
17+
agent: "codex"
18+
shell_pid: "6888"
19+
review_status: "approved"
20+
reviewed_by: "Mike Gifford"
1821
history:
19-
- timestamp: "2026-02-21T20:12:31Z"
20-
lane: "planned"
21-
agent: "system"
22-
shell_pid: ""
23-
action: "Prompt generated via /spec-kitty.tasks"
22+
- timestamp: '2026-02-21T20:12:31Z'
23+
lane: planned
24+
agent: system
25+
shell_pid: ''
26+
action: Prompt generated via /spec-kitty.tasks
2427
---
2528

2629
# Work Package Prompt: WP01 – Pipeline Foundation and Configuration
@@ -104,3 +107,7 @@ history:
104107
## Activity Log
105108

106109
- 2026-02-21T20:12:31Z – system – lane=planned – Prompt generated.
110+
- 2026-02-21T20:15:30Z – codex – shell_pid=6888 – lane=doing – Assigned agent via workflow command
111+
- 2026-02-21T20:29:39Z – codex – shell_pid=6888 – lane=for_review – Ready for review: foundation scaffold, config validation, ingest normalization, and tests complete
112+
- 2026-02-21T20:30:41Z – codex – shell_pid=6888 – lane=doing – Started review via workflow command
113+
- 2026-02-21T20:36:47Z – codex – shell_pid=6888 – lane=done – Review passed: WP01 foundation implemented cleanly; unit tests pass (8/8); dependency check N/A (none); dependent WP02 remains planned; dependency declarations align with current module coupling and staged architecture.

‎kitty-specs/002-daily-dap-quality-benchmarking/tasks/WP02-scanner-execution-layer.md‎

Lines changed: 28 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,30 @@
11
---
2-
work_package_id: "WP02"
3-
subtasks:
4-
- "T006"
5-
- "T007"
6-
- "T008"
7-
- "T009"
8-
- "T010"
9-
title: "Scanner Execution Layer"
10-
phase: "Phase 2 - Data Collection"
11-
lane: "planned"
2+
work_package_id: WP02
3+
title: Scanner Execution Layer
4+
lane: "done"
125
dependencies:
13-
- "WP01"
14-
assignee: ""
15-
agent: ""
16-
shell_pid: ""
17-
review_status: ""
18-
reviewed_by: ""
6+
- WP01
7+
base_branch: 002-daily-dap-quality-benchmarking-WP01
8+
base_commit: f14ce0e83305e07ec1fbe4db2a5030cff44d6485
9+
created_at: '2026-02-21T20:37:59.357606+00:00'
10+
subtasks:
11+
- T006
12+
- T007
13+
- T008
14+
- T009
15+
- T010
16+
phase: Phase 2 - Data Collection
17+
assignee: ''
18+
agent: "codex"
19+
shell_pid: "3570"
20+
review_status: "approved"
21+
reviewed_by: "Mike Gifford"
1922
history:
20-
- timestamp: "2026-02-21T20:12:31Z"
21-
lane: "planned"
22-
agent: "system"
23-
shell_pid: ""
24-
action: "Prompt generated via /spec-kitty.tasks"
23+
- timestamp: '2026-02-21T20:12:31Z'
24+
lane: planned
25+
agent: system
26+
shell_pid: ''
27+
action: Prompt generated via /spec-kitty.tasks
2528
---
2629

2730
# Work Package Prompt: WP02 – Scanner Execution Layer
@@ -106,3 +109,7 @@ history:
106109
## Activity Log
107110

108111
- 2026-02-21T20:12:31Z – system – lane=planned – Prompt generated.
112+
- 2026-02-21T20:37:59Z – codex – shell_pid=6888 – lane=doing – Assigned agent via workflow command
113+
- 2026-02-21T21:55:08Z – codex – shell_pid=6888 – lane=for_review – Ready for review: implemented T006-T010 scanner execution layer (Lighthouse + ScanGov runners, per-URL normalization, bounded retry/timeout execution manager, and status/diagnostics), with unit tests passing (13/13).
114+
- 2026-02-21T21:57:13Z – codex – shell_pid=3570 – lane=doing – Started review via workflow command
115+
- 2026-02-21T21:58:31Z – codex – shell_pid=3570 – lane=done – Review passed: WP02 scanner execution layer satisfies T006-T010; tests pass (13/13); dependency WP01 is on main; dependent WP03 is planned; dependency declarations match staged coupling boundaries.

0 commit comments

Comments
 (0)