Add CIMIS weather data integration#253
Open
dmpayton wants to merge 19 commits into
Open
Conversation
Add DewPoint, SoilTemperature, WindSpeed, WindDirection, Precipitation, SolarRadiation, NetRadiation, VaporPressure, ETo, and ETr entry models. DewPoint and SoilTemperature include fahrenheit/celsius conversion properties. Includes comprehensive test coverage for all new entry types.
- Add missing get_monitor_id to CIMISAdmin.readonly_fields to prevent a 500 on the admin change page (fieldsets reference it as a field). - Remove dead fields = MonitorAdmin.fields line in CIMISAdmin (MonitorAdmin defines fieldsets, not fields, so this was a no-op). - Rename CIMIS_APP_KEY setting to CIMIS_API_KEY throughout (settings, api.py) to match the key name already configured in .env, and add a missing-key guard to discover_cimis_stations matching the pattern used in airnow/tasks.py. - Remove unused Decimal, ROUND_HALF_UP, LevelSet, AQLevel imports from entries/models/base.py (only used by subclasses, which import them independently). - Add a data_items assertion to ImportCimisDataTests.test_calls_api_with_all_known_station_numbers.
The client was built against CIMIS's publicly documented REST API (https://et.water.ca.gov/api/{data,station} with appKey as a query param), but live testing with a real API key showed CIMIS's WAF now rejects that scheme. CIMIS has migrated to an Azure API Management gateway. Live testing against the new scheme confirmed it works and returns data in the same shape handle_payload() already expects: - Base URL is https://et.water.ca.gov (no /api suffix). - Auth is the Ocp-Apim-Subscription-Key HTTP header, not a query param; set once on the session so every request carries it. - GET /StationWeb/GetAllStations replaces GET /api/station, no query params needed. - GET /StationWeb/GetDataByStationNumber replaces GET /api/data, with the "targets" param renamed to "stationNbrs" and a new required "isHourly": "true" param. Updated CIMISAPITests to assert the new URLs, the renamed/added params, the absence of appKey from the query string, and the subscription-key header. Confirmed the updated tests fail against the old implementation (real red) before verifying they pass against the fix.
import_cimis_data only re-queries the current calendar day, so hours that finalize after CIMIS's QC lag window closes (especially the 2400 boundary hour) can be missed once the day rolls over. Add finalize_cimis_data, a daily task that re-pulls yesterday's full day once, after CIMIS's typical QC window has closed. Safe to re-run: entries are deduplicated on (monitor, timestamp, sensor, stage, processor), so this only fills genuine gaps. Extracted the shared station-list/API-call/record-iteration logic from import_cimis_data into _ingest_cimis_data(target_date), parameterized by date, used by both tasks.
Adds California Department of Water Resources to the About page's Data Providers section (logo links to CIMIS, the specific data product SJVAir integrates with, matching the existing CARB/AQview pattern). Adds a new "Weather & Climate Data" category to the Integrations page with a CIMIS entry describing what it provides. DWR's source logo was 5068x844px (385KB); resized to 900px wide to match the size of sibling agency logos in this directory.
_ingest_cimis_data now resolves all known CIMIS monitors into a station_number -> monitor map once per run, and passes the resolved monitor into process_cimis_data instead of having it re-query per record. With ~15 SJV stations reporting up to 24 hourly records/day, this was up to 300+ redundant lookups per hourly run by end-of-day. process_cimis_data still resolves its own monitor when called without one, so it remains correct as a standalone task. Also guard against malformed records from the API: process_cimis_data catches KeyError/ValueError/TypeError/AttributeError from handle_payload (e.g. a record missing 'Date'/'Hour') and reports to Sentry instead of raising, so one bad record doesn't abort the rest of the batch. process_cimis_station similarly skips a station payload missing 'StationNbr' instead of raising an IntegrityError from inserting a null station_number.
MonitorFilter.filter_device() in the v2 API hardcodes a device-name -> lookup-field map for the ?device= query param. AirGradient was already missing; CIMIS was missing too since it's new. Neither raised an error - the filter method just no-ops and returns the unfiltered queryset, so ?device=CIMIS silently returned every monitor instead of just CIMIS ones. v1's MonitorFilter has the same pattern and is also missing AQview, AirGradient, and CIMIS, but that's a much older, pre-existing gap unrelated to this branch - left alone here.
CIMISAdmin listed station_number in readonly_fields, but the base MonitorAdmin.fieldsets it inherited unchanged never references that field - readonly_fields only controls editability of fields already present in a fieldset, it isn't an independent display mechanism. The result: CIMIS's one truly identifying field (its external station number) never actually rendered on the change form. Confirmed via a live admin request before and after the fix. Also surface it in the changelist (list_display) and make it searchable (search_fields), matching how LCSMonitorAdmin does the same for PurpleAir/AirGradient's sensor_id. The admin menu itself needs no changes: no custom AdminSite, no admin-reordering package, and the cimis app is already alphabetically placed in INSTALLED_APPS. Its menu label will read "Cimis" (Django's default title-casing of the app label), consistent with how "Airnow" and "Aqview" already render - not a new/CIMIS-specific issue.
ClosestMonitor, CurrentData, and MonitorsAt had no filter_class wired in, so ?device=CIMIS (or any MonitorFilter param) was silently ignored on those three endpoints, unlike MonitorList. Wiring in filter_class = MonitorFilter directly doesn't work here, for two different reasons depending on the endpoint: - All three call with_latest_entry()/with_entry_as_of(), which fetch the queryset and mutate each instance in place (setattr for `latest_entry`, `latest_<type>`) rather than adding a lazy DB annotation. filter_class makes the base ListModelMixin call filter_queryset() a second time after get_queryset() returns - a plain .filter() call that clones the queryset and silently drops those in-memory mutations, since they were never part of the SQL annotation to begin with. - ClosestMonitor additionally slices to the 3 nearest before that second filter call would run, so filtering after the slice would wrongly exclude a matching monitor just outside the raw top 3. - MonitorsAt's with_entry_as_of() returns a plain list, not a queryset at all, so a second .filter() call on it would raise outright rather than just losing data. Fixed by applying MonitorFilter manually inside each get_queryset(), before with_latest_entry()/with_entry_as_of()/the top-3 slice, and not setting filter_class at all so the base class's automatic second pass is a no-op. Caught by directly reproducing the underlying exception (patched server_error to re-raise instead of returning a masked 500) rather than assuming filter_class was a safe drop-in for every list endpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CIMISmonitor type that pulls hourly weather station data (temp, humidity, dew point, wind, precipitation, solar/net radiation, vapor pressure, soil temp, ETo/ETr) from California's CIMIS API for San Joaquin Valley countiesentriesapp (reusable by future weather-data providers, not CIMIS-specific)entries/models.pyinto a package (base/particulates/meteorological/gases) since this doubled the size of the meteorological sectionapi.py/*APInaming convention (renamesairnow/client.py→api.py)python-CIMISPyPI package — seedocs/superpowers/specs/2026-07-11-cimis-integration-design.mdfor the investigation (stale GitHub, untracked PyPI releases). Hand-rolled a minimal client instead.StationWeb/GetDataByStationNumber, header-based auth), not the stale query-param scheme CIMIS's own public docs still describe — confirmed via live testing with a real API key, since the documented scheme is now WAF-rejected.Test plan
readonly_fieldsbug fixed, missing API-key guard added, migration hygiene cleaned up)CIMISAPIclient verified end-to-end against the live CIMIS API (274 real stations, real hourly records)CIMIS_API_KEYenv var in deploy config (not yet set in production)