Skip to content

feat(vozbox): integrate VOZbox air quality monitors (CCEJN)#250

Open
dmpayton wants to merge 21 commits into
mainfrom
feature/vozbox
Open

feat(vozbox): integrate VOZbox air quality monitors (CCEJN)#250
dmpayton wants to merge 21 commits into
mainfrom
feature/vozbox

Conversation

@dmpayton

@dmpayton dmpayton commented Jul 12, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds `VOZBox` monitor type for CCEJN's VOZbox devices, which upload CSV data to `QuinnResearch/carbVoz_data` on GitHub
  • Implements `VozBoxClient` context manager to download and parse daily/calibration CSVs via the GitHub Contents API and `raw.githubusercontent.com`
  • Full LCS PM2.5 pipeline (RAW → CORRECTED → CLEANED → CALIBRATED), dual-channel PM1/PM2.5/PM10, temperature, humidity, and O3
  • `O3_VOZBox` calibration processor stub (inactive until a `DefaultCalibration` DB record is created and a model is trained)
  • `import_realtime` periodic Huey task (every 10 min) + `process_device` per-device task with dedup and 10-minute downsampling
  • `import_vozbox_cal` management command for backfilling externally-calibrated O3 from Quinn Research's cal CSVs
  • Django admin with `sensor_id` in list/search/fieldset; add permission disabled (monitors are auto-created on import)
  • VOZbox entry in admin sidebar menu
  • CCEJN added to the about page Data Providers section
  • `VOZBoxHealthCheck` registered at `/system-status/data-feeds/` (30-minute staleness limit)
  • `?device=VOZBox` filter supported in both v1 and v2 monitor APIs

Key design decisions

  • `VOZBox` inherits `Monitor` directly (not `LCSMixin`) — the device's `coreid` is a 24-char hex string, incompatible with `LCSMixin.sensor_id` (IntegerField)
  • O3 `default_stage=CALIBRATED` so external cal entries (with `processor=''`) surface in `LatestEntry` before the internal calibration model is trained
  • `_bin_rows()` downsamples incoming rows to 10-minute buckets (earliest row per bucket) before scheduling `process_device` tasks, keeping Redis payloads small even for 1Hz devices
  • PM values > 9,999 are dropped (one device uploads bogus readings that overflow the field)
  • `get_or_create` used for race-condition-safe monitor creation across concurrent Huey workers
  • Supports `GITHUB_API_TOKEN` env var to raise GitHub API rate limits

Test plan

  • 41 tests across `VozBoxClientParseTests`, `VozBoxClientHTTPTests`, `VOZBoxModelTests`, `O3VOZBoxProcessorTests`, `ProcessDeviceTests`, `BinRowsTests`, `ImportVozboxCalTests`
  • `docker compose run --rm test pytest camp/apps/monitors/vozbox/tests.py -v` — 41 passed
  • Manual import confirmed working against live `QuinnResearch/carbVoz_data` (13 devices active)

dmpayton added 21 commits June 29, 2026 19:19
Covers client library, monitor model, tasks, O3 calibration processor stub, calibrated backfill management command, admin, and testing plan.
- Rename VOZBOX_GITHUB_TOKEN → GITHUB_API_TOKEN
- Expand O3 calibration section: external (processor='') vs internal (future VOZBox_O3_xyz)
  and how they interact with LatestEntry
- Remove inferred Quinn Research formula; mark as TBD pending confirmation
8 tasks: scaffold, VozBoxClient (CSV + HTTP), VOZBox model, O3_VOZBox processor,
tasks, admin, import_vozbox_cal management command. Full TDD throughout.
- Add CCEJN to datafiles/data-providers.yaml with NOAA-sized classes
- Improve VOZBoxAdmin: sensor_id in fieldset as readonly, disable manual add
- Add VOZbox entry to admin menu template
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant