Implement Site Scanning alert action - #1
Open
ethangardner wants to merge 1 commit into
Open
Conversation
Closes GSA/site-scanning#1972 Complete GitHub Action for monitoring federal websites using Site Scanning data. Features: - Change-detection (latest vs previous) + state-checking (bad current values) - Two watchlist models (exact domain / base: for all subdomains) - Noise suppression calibrated against real data - Fingerprinted rolling issues (no spam) - Zero runtime dependencies (Python 3.9+ stdlib only) - 34 unit tests + CI docs enforcement 17 files, 2,175 insertions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the GSA Site Scanning Alert composite Action: a scheduled workflow that downloads the daily Site Scanning CSV snapshots, filters them to a watchlist of monitored domains, evaluates configurable alert rules, and files/updates a GitHub issue when something needs attention.
Why
Teams that publish federal websites want to know quickly when Site Scanning data shows a status regression (site down, cert error, scan failure) for the domains they own, without having to manually check the dataset. This gives them a drop-in GitHub Action + daily workflow that turns that dataset into actionable, deduplicated GitHub issues.
What's included
scripts/snapshot.py— downloads and parses the latest/previous Site Scanning CSV snapshots with column projection, retry/backoff, freshness checks, and rotation detection (so a workflow re-run mid-day is a no-op).scripts/rules.py— two alert modes:change(diff latest vs. previous snapshot on configurable fields) andstate(flag bad current values like 5xx status codes or non-live sites), with noise-suppression knobs (ignore_blank_transitions,ignore_transitions) and amax_changessummary fallback for noisy days.scripts/issues.py— fingerprinted rolling-issue lifecycle: creates an issue on first alert, comments when the alert set changes, no-ops on identical re-runs, and (optionally) comments when a condition clears. Avoids duplicate issue spam across daily runs.scripts/site_scanning_alerts.py— entrypoint that wires the above together fromaction.ymlinputs, with dry-run support and step-summary output.action.yml— composite Action definition exposing all the above as configurable inputs (watchlist path, mode, fields, thresholds, labels, dry-run, etc.)..github/workflows/site-scanning-alerts.yml— example daily cron workflow (30 min after the 15:00 UTC snapshot rotation) plus a manualworkflow_dispatchdry-run..github/workflows/test.yml— CI running the unit test suite and a doc-consistency check on every PR/push tomain.watchlist.txt— documented, ready-to-edit watchlist supporting both exact-domain andbase:domain.govmatching.tests/— unit coverage for snapshot filtering/freshness/rotation, rule evaluation, issue fingerprinting, and a doc test that keeps the README's input table in sync withaction.yml.primary_scan_statustransitions.Testing
python3 -m unittest discover tests -v— all tests pass.tests/fixtures/*.csv.Next in the stack
#2 hardens this implementation (TLS verification, alert-lifecycle correctness, pagination) based on review of this PR — see that PR for details.