You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
daily/aashto: track per-year Crash.csv, sync cc2mc2mn.json dep hash
Root-cause the aashto-stage rebuild-loop that broke today's daily.
Three latent brittle spots collided when my earlier pull-list expansion
forced DVX to walk upstream instead of stopping at the target stage:
1. **Per-year `Crash.csv` was on disk only (not DVC-tracked)** for
2023/2024/2025. `dvx add` all three (~2 GB total) and push. Each
`Crash.csv.dvc` md5 matches the corresponding `crashes.parquet.dvc`
dep hash, so downstream stages see the dep as satisfied without
needing `normalize.py` to re-run.
2. **Daily didn't pull `Crash.csv.dvc`** even though it's an upstream
dep of the per-year `crashes.parquet` stage — pulls added.
3. **`cc2mc2mn.json` was updated 2026-07-04 (`ca5cef2b69a`) without
refreshing the aashto-chain .dvc dep hashes**. Bumped in the four
affected files:
- `aashto_combined_crashes.parquet.dvc`
- `aashto_supplemented_occupants.parquet.dvc`
- `aashto_supplemented_pedestrians.parquet.dvc`
- `aashto_supplemented_vehicles.parquet.dvc`
Output md5 is unchanged (verified on disk) — this is metadata sync,
no recompute.
Also expand daily.yml comment above the per-year pulls to explain the
`crashes.parquet.dvc` + `Crash.csv.dvc` pairing (was #127-adjacent).
Prior daily runs (07-05 → 07-07) succeeded despite this staleness only
because DVX's plan stopped at the direct target (`njsp_njdot_match`,
already up-to-date). Today's plan walks 4 levels due to more per-year
outputs joining the pull list, exposing the stale chain.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0 commit comments