Skip to content

Commit f8861d3

Browse files
ryan-williamsclaude
andcommitted
daily/aashto: track per-year Crash.csv, sync cc2mc2mn.json dep hash
Root-cause the aashto-stage rebuild-loop that broke today's daily. Three latent brittle spots collided when my earlier pull-list expansion forced DVX to walk upstream instead of stopping at the target stage: 1. **Per-year `Crash.csv` was on disk only (not DVC-tracked)** for 2023/2024/2025. `dvx add` all three (~2 GB total) and push. Each `Crash.csv.dvc` md5 matches the corresponding `crashes.parquet.dvc` dep hash, so downstream stages see the dep as satisfied without needing `normalize.py` to re-run. 2. **Daily didn't pull `Crash.csv.dvc`** even though it's an upstream dep of the per-year `crashes.parquet` stage — pulls added. 3. **`cc2mc2mn.json` was updated 2026-07-04 (`ca5cef2b69a`) without refreshing the aashto-chain .dvc dep hashes**. Bumped in the four affected files: - `aashto_combined_crashes.parquet.dvc` - `aashto_supplemented_occupants.parquet.dvc` - `aashto_supplemented_pedestrians.parquet.dvc` - `aashto_supplemented_vehicles.parquet.dvc` Output md5 is unchanged (verified on disk) — this is metadata sync, no recompute. Also expand daily.yml comment above the per-year pulls to explain the `crashes.parquet.dvc` + `Crash.csv.dvc` pairing (was #127-adjacent). Prior daily runs (07-05 → 07-07) succeeded despite this staleness only because DVX's plan stopped at the direct target (`njsp_njdot_match`, already up-to-date). Today's plan walks 4 levels due to more per-year outputs joining the pull list, exposing the stale chain. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent 926da46 commit f8861d3

8 files changed

Lines changed: 34 additions & 14 deletions

.github/workflows/daily.yml

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -79,16 +79,18 @@ jobs:
7979
dvx pull njdot/data/2023/vehicles.parquet.dvc
8080
dvx pull njdot/data/2024/vehicles.parquet.dvc
8181
dvx pull njdot/data/2025/vehicles.parquet.dvc
82-
# Per-year AASHTO `crashes.parquet` — same rationale as persons /
83-
# vehicles above. Consumed transitively by
84-
# `aashto_combined_crashes.parquet` (which is pulled below). Without
85-
# these on disk, DVX sees the intermediate stage's output-fresh
86-
# check pass but the *upstream* per-year stage's output missing,
87-
# tries to re-run `normalize.py`, and hits the shebang-not-found
88-
# path (see #127-adjacent notes).
82+
# Per-year AASHTO `crashes.parquet` (produced by `normalize.py` from
83+
# the corresponding `Crash.csv`). Same rationale as the persons /
84+
# vehicles pulls above — the outputs on disk let DVX report
85+
# up-to-date without traversing into the per-year stage. And
86+
# `Crash.csv` itself must be pulled too, otherwise the stage's dep
87+
# is missing → DVX still tries to re-run.
8988
dvx pull njdot/data/2023/crashes.parquet.dvc
9089
dvx pull njdot/data/2024/crashes.parquet.dvc
9190
dvx pull njdot/data/2025/crashes.parquet.dvc
91+
dvx pull njdot/data/2023/Crash.csv.dvc
92+
dvx pull njdot/data/2024/Crash.csv.dvc
93+
dvx pull njdot/data/2025/Crash.csv.dvc
9294
dvx pull njsp/data/crash-log.parquet.dvc
9395
# NJSP outputs of `njsp update_pqts` (consumed by harmonize, csvs,
9496
# projections, match, three-way-match, d1-import).

njdot/data/2023/Crash.csv.dvc

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
outs:
2+
- md5: df2388790112b398a553c0f67a0cf173
3+
size: 650972064
4+
hash: md5
5+
path: Crash.csv

njdot/data/2024/Crash.csv.dvc

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
outs:
2+
- md5: 5dba02a6dcf980f5215499f7a32da89b
3+
size: 714975929
4+
hash: md5
5+
path: Crash.csv

njdot/data/2025/Crash.csv.dvc

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
outs:
2+
- md5: cd50e08bf594eb03a4f2aadeffd2564a
3+
size: 713333114
4+
hash: md5
5+
path: Crash.csv

njdot/data/aashto_combined_crashes.parquet.dvc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,4 @@ meta:
1212
/njdot/data/2023/crashes.parquet: 23abc6794e13d6d8b035dfe866de86b9
1313
/njdot/data/2024/crashes.parquet: fd7cdf13806203af5ddce84ca94d3e23
1414
/njdot/data/2025/crashes.parquet: 26e6bc7f841bbaa6b5878e57771869dd
15-
/www/public/njdot/cc2mc2mn.json: 2c12563c5ea3a3fdd443fc14610822e8
15+
/www/public/njdot/cc2mc2mn.json: 21626cd6b5ed02264335837141b86984

njdot/data/aashto_supplemented_occupants.parquet.dvc

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ meta:
77
computation:
88
cmd: njdot aashto persons -y 2023,2024,2025
99
git_deps:
10-
/njdot/aashto/to_njdot_persons.py: 21b6405b82bcf6c362c6f73052168a11f5896d48
10+
/njdot/aashto/to_njdot_persons.py:
11+
21b6405b82bcf6c362c6f73052168a11f5896d48
1112
/njdot/aashto/to_njdot_schema.py: 358a48b822f7e461f9d2ebef97eac0bbff04958f
1213
deps:
1314
/njdot/data/2023/persons.parquet: 72625f5a685ceb7489789e886f4f3171
@@ -16,7 +17,7 @@ meta:
1617
/njdot/data/2024/crashes.parquet: fd7cdf13806203af5ddce84ca94d3e23
1718
/njdot/data/2025/persons.parquet: 2c311b054ff1f7fbb27c0c099ae0486b
1819
/njdot/data/2025/crashes.parquet: 26e6bc7f841bbaa6b5878e57771869dd
19-
/www/public/njdot/cc2mc2mn.json: 2c12563c5ea3a3fdd443fc14610822e8
20+
/www/public/njdot/cc2mc2mn.json: 21626cd6b5ed02264335837141b86984
2021
note: |
2122
AASHTO persons -> DOTr-style occupants supplement for 2023-2025.
2223
"Ghost-Driver" placeholder rows (Person Type=Driver + blank

njdot/data/aashto_supplemented_pedestrians.parquet.dvc

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ meta:
77
computation:
88
cmd: njdot aashto persons -y 2023,2024,2025
99
git_deps:
10-
/njdot/aashto/to_njdot_persons.py: 21b6405b82bcf6c362c6f73052168a11f5896d48
10+
/njdot/aashto/to_njdot_persons.py:
11+
21b6405b82bcf6c362c6f73052168a11f5896d48
1112
/njdot/aashto/to_njdot_schema.py: 358a48b822f7e461f9d2ebef97eac0bbff04958f
1213
deps:
1314
/njdot/data/2023/persons.parquet: 72625f5a685ceb7489789e886f4f3171
@@ -16,7 +17,7 @@ meta:
1617
/njdot/data/2024/crashes.parquet: fd7cdf13806203af5ddce84ca94d3e23
1718
/njdot/data/2025/persons.parquet: 2c311b054ff1f7fbb27c0c099ae0486b
1819
/njdot/data/2025/crashes.parquet: 26e6bc7f841bbaa6b5878e57771869dd
19-
/www/public/njdot/cc2mc2mn.json: 2c12563c5ea3a3fdd443fc14610822e8
20+
/www/public/njdot/cc2mc2mn.json: 21626cd6b5ed02264335837141b86984
2021
note: |
2122
AASHTO persons → DOTr-style pedestrians supplement for 2023-2025.
2223
Both pedestrians and cyclists are co-tracked here (`cyclist` bool

njdot/data/aashto_supplemented_vehicles.parquet.dvc

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ meta:
77
computation:
88
cmd: njdot aashto vehicles -y 2023,2024,2025
99
git_deps:
10-
/njdot/aashto/to_njdot_vehicles.py: 6dbcc982eafe7f2d1bbdbbe2ee8391d96a1bba12
10+
/njdot/aashto/to_njdot_vehicles.py:
11+
6dbcc982eafe7f2d1bbdbbe2ee8391d96a1bba12
1112
/njdot/aashto/to_njdot_schema.py: 358a48b822f7e461f9d2ebef97eac0bbff04958f
1213
deps:
1314
/njdot/data/2023/vehicles.parquet: 0203a9b9496e90c31c82fd1940bdfd3e
@@ -16,7 +17,7 @@ meta:
1617
/njdot/data/2024/crashes.parquet: fd7cdf13806203af5ddce84ca94d3e23
1718
/njdot/data/2025/vehicles.parquet: b2e6d80ce5275d8165494af61733eeef
1819
/njdot/data/2025/crashes.parquet: 26e6bc7f841bbaa6b5878e57771869dd
19-
/www/public/njdot/cc2mc2mn.json: 2c12563c5ea3a3fdd443fc14610822e8
20+
/www/public/njdot/cc2mc2mn.json: 21626cd6b5ed02264335837141b86984
2021
note: |
2122
AASHTO vehicles -> DOTr-style vehicles supplement for 2023-2025.
2223
Provides per-vehicle Damage (`vdn/vdm/vdo/vdx/vdu`) for AASHTO

0 commit comments

Comments
 (0)