Add SJVAPCD daily air quality forecast integration#255
Open
dmpayton wants to merge 20 commits into
Open
Conversation
…zone.now() in tests issued_date must reflect when THIS SERVER pulled the feed, not the feed's self-reported lastBuildDate, since the delete-and-recreate idempotency check is keyed on it. Revert fetch_forecasts() to compute issued_date from timezone.now() as the plan specifies, and fix the tests to mock timezone.now() instead of depending on the real wall-clock date.
…oesn't abort the whole run Wrap each item's parse+create body in fetch_forecasts() in a try/except (StopIteration, ValueError, AttributeError). Previously an unexpected feed shape for a single zone (e.g. "Unavailable" AQI text, a missing pubdate, or a missing airAlertStatus element) would propagate out of the enclosing transaction.atomic() block and roll back forecasts for all eight counties, not just the one affected zone.
Reuse the existing ReadOnlyAdminMixin instead of hand-rolled permission overrides, and add search, filters, and an optimized queryset so the Forecasts section is fully browsable/searchable from the admin index.
Follows the existing MonitorHealthCheck pattern (camp/apps/monitors/health_checks.py): warns if no Forecast rows exist, or if the most recent one is older than 27 hours (the feed's ~24h cadence plus the fetch_forecasts cron window's slack). Registered on the existing /system-status/data-feeds/ page.
…failures - Wrap each zone's writes in a nested transaction.atomic() savepoint and catch django.db.Error alongside the existing parse-error types, so a DB-level failure (e.g. a value that violates a field constraint) for one zone no longer poisons the outer transaction and rolls back every other already-processed zone. - Log the skipped-zone case at ERROR instead of WARNING. Sentry's default logging integration only creates an issue at ERROR+ (WARNING is just a breadcrumb), so a persistent feed problem for one zone now actually alerts instead of silently degrading that zone's data forever.
… page Follows the admin_change_link() convention already used in camp/apps/pesticides/admin.py.
…I_LEVELS Moves the inline Levels = LevelSet(...) blocks out of PM25/PM100/CO/NO2/O3/SO2 (camp/apps/entries/models.py) into named constants in levels.py -- pure code motion, no behavior change (91 existing entries/alerts/regions/monitors tests still pass unchanged). Adds AQI_LEVELS: a new LevelSet for the EPA AQI index itself (0-500), as opposed to a pollutant's raw concentration -- something the existing per-pollutant LevelSets can't represent, since their breakpoints are in each pollutant's native unit. Breakpoints match camp.utils.aqi.aqi_label. Wires this into forecasts: Forecast.color derives a hex color from aqi_value via AQI_LEVELS.get_color(), matching the blended-color convention already used for monitor markers (camp/apps/regions/admin.py). Exposed on the API serializer so the frontend map layer can shade counties without reimplementing the AQI color scale.
Renames PM25_LEVELS -> levels.PM25, AQI_LEVELS -> levels.AQI, etc. Both camp.apps.entries.models and camp.apps.forecasts.models now do `from . import levels` / `from camp.apps.entries import levels` and reference sets as levels.PM25, levels.AQI, etc., instead of importing each name individually -- avoids colliding with the identically-named entry model classes (PM25, O3, ...) since the short names now only exist as attributes on the levels module, never imported into local scope.
…p SVG Adds camp.apps.regions.forecast_zones + the import_forecast_zones command, which fit an affine transform from the forecast map's SVG pixel space to lat/lon (using the 6 counties that map 1:1 to the feed's zones as ground-control points -- validated to IoU >= 0.99 against their real boundaries) and use it to derive real geometry for the 3 zones that don't: - Kern (SJV Air Basin portion): real Kern County intersected with the transformed SVG shape (SJVAPCD's zone excludes Kern's desert/mountain portion). - Tulare (SJV Valley portion): same approach against real Tulare County. - Sequoia National Park and Forest: real Tulare County minus the Tulare zone above. Verified this overlaps ~100% of the SVG-derived Sequoia shape and ~0% of Kern's excluded portion -- i.e. Sequoia is carved entirely from Tulare, not Kern -- so Tulare and Sequoia now tile the real county exactly, with no gap or overlap between them. All 3 are imported as Region(type=CUSTOM) records with metadata recording their derivation. The forecasts app's zone-to-region mapping now points Kern and Tulare at these instead of their political counties, and Sequoia (previously dropped entirely) is now included -- every zone in the feed maps to a region, so a daily pull now creates 18 Forecast rows (9 zones x 2 horizons) instead of 16. Source SVG saved at datafiles/sjvapcd-forecast-areas.svg (cleaned: just the 9 named region outlines, no fonts/colors/labels/tooltip markup).
RegionSerializer has no type-based branching, so custom regions (Kern SJV Air Basin, Tulare valley, Sequoia) already got full boundary GeoJSON through /api/2.0/forecasts/ -- verified live against the real imported geometry. This locks that behavior in with an explicit regression test.
… spec
Both are legitimately optional -- burn_el.get('status', '') and
burn_el.text or '' in the ingestion task can produce empty strings.
…etup The original spec's "Out of Scope" note said accurate Kern/Sequoia zone geometry was deferred with county boundaries as an approximation -- that was superseded once the SVG-derivation approach shipped on this same branch. Added an addendum explaining what actually shipped, and a new CLAUDE.md section documenting the one-time import_forecast_zones setup step (parallel to the existing Pesticide Data Import section).
Covers the SVG path parser, affine fit, transform, and IoU helpers with plain synthetic-data unit tests, plus integration tests against the real SVG + regions fixture: all 9 shapes load, the ground-control fit validates, the derived Kern/Tulare/Sequoia geometry is valid and closely reproduces what's already imported in fixtures/regions.yaml, Tulare and Sequoia tile the real county with no gap or overlap, and a deliberately scrambled ground-control mapping correctly trips the IoU validation gate. Also generalizes region_boundary_shape() to accept a region_type so tests can load the CUSTOM regions' boundaries the same way, not just COUNTY.
- requirements/base.txt: restore alphabetical order (shapely was inserted before setuptools instead of after) - forecast_zones.py / design spec: correct the ground-control IoU claim from ">= 0.99" to ">= 0.98" -- the actual measured minimum (san-joaquin, 0.9898) was technically just under the originally stated threshold
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ingestion
camp.apps.forecastsapp ingesting SJVAPCD's daily air-quality-forecast feed (https://ww2.valleyair.org/aqinfo/airstatus.xml) into aForecastmodel, via a daily Huey periodic task (fetch_forecasts) plus a manual management command wrapper.burnStatus:/AQI:XML namespace prefixes resolve to the same URI, so same-horizon elements share one Clark-notation tag and have to be told apart by content shape rather than by tag lookup.Accurate zone geometry (not just county approximations)
camp.apps.regions.forecast_zonesderives real lat/lon geometry for these 3 zones from a locally-saved copy of the forecast map's SVG (datafiles/sjvapcd-forecast-areas.svg): fits an affine transform from the SVG's pixel space to lat/lon using the 6 zones that do match a county as ground-control points (validated to IoU ≥ 0.98 against their real boundaries), then uses real county boundaries for the outer edges and the transform only for the internal dividing line. Verified Sequoia is carved entirely from Tulare (not Kern) by checking geometric overlap against each county's excluded portion.import_forecast_zonescommand asRegion(type=CUSTOM)+Boundaryrecords — see the new "SJVAPCD Forecast Setup" section in CLAUDE.md for the required one-time setup step per environment.Forecastrows instead of 16.API
GET /api/2.0/forecasts/(list + detail) endpoints, filterable byregion_id/forecast_date/issued_date, defaulting to current+future forecasts. Each row nests the region's boundary GeoJSON (via the existingRegionSerializer) — verified this works identically for the new custom regions, not just counties — so a frontend map layer doesn't need a second request.Forecast.colorderives a hex color from the AQI value via a newlevels.AQIscale, matching the blended-color convention already used for monitor markers elsewhere in the app.Supporting refactor
LevelSetdefinitions out ofcamp.apps.entries.modelsintocamp.apps.entries.levels(pure code motion, verified value-for-value identical), and addedlevels.AQI— a scale for the EPA AQI index itself, which none of the existing pollutant-specific sets could represent.Admin & ops
Forecastis fully browsable (search, filters, region link, optimized queryset) but stays read-only, since the data is fully machine-ingested daily./system-status/data-feeds/health check page — warns if no forecasts exist or the most recent one is older than 27 hours.Design spec:
docs/superpowers/specs/2026-07-12-sjvapcd-forecast-design.md(includes a 2026-07-13 addendum on the zone geometry work)Implementation plan:
docs/superpowers/plans/2026-07-12-sjvapcd-forecast.mdTest plan
docker compose run --rm test pytest— 666 passed), including new unit + integration coverage for the SVG parsing/affine-fit/geometry-derivation math