Skip to content

Commit ea32628

Browse files
committed
daily CI: pull person supplements + master O/P/V; drop persons stage
`supplement_with_sp.py` now reads `aashto_supplemented_{occupants, pedestrians}.parquet` for VTC; `cmymc.py` reads master `vehicles/ occupants/pedestrians.parquet`. CI's prep step now pulls all of these from S3. Person supplements aren't rebuilt in CI: their per-year `persons.parquet` inputs are side-effects of `normalize.py`, which only runs locally after a manual `Crash.csv` download. Pulling from S3 is sufficient (they only change when the user refreshes Crash.csv).
1 parent d44df44 commit ea32628

1 file changed

Lines changed: 10 additions & 3 deletions

File tree

.github/workflows/daily.yml

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,8 +55,13 @@ jobs:
5555
- name: Pull DVC-tracked deps from S3
5656
run: |
5757
dvx pull njdot/data/crashes.parquet
58+
dvx pull njdot/data/vehicles.parquet.dvc
59+
dvx pull njdot/data/occupants.parquet.dvc
60+
dvx pull njdot/data/pedestrians.parquet.dvc
5861
dvx pull njdot/data/*/NewJersey*Accidents.pqt.dvc
5962
dvx pull njdot/data/aashto_combined_crashes.parquet.dvc
63+
dvx pull njdot/data/aashto_supplemented_occupants.parquet.dvc
64+
dvx pull njdot/data/aashto_supplemented_pedestrians.parquet.dvc
6065
dvx pull njsp/data/crash-log.parquet.dvc
6166
dvx pull www/public/Municipal_Boundaries_of_NJ.geojson
6267
- name: DVX status
@@ -91,9 +96,11 @@ jobs:
9196
- name: Supplement AASHTO with NJSP-only residuals
9297
if: ${{ !inputs.targets }}
9398
run: $DVX njdot/data/aashto_supplemented_crashes.parquet.dvc
94-
- name: Supplement AASHTO persons (occupants + pedestrians)
95-
if: ${{ !inputs.targets }}
96-
run: $DVX njdot/data/aashto_supplemented_occupants.parquet.dvc njdot/data/aashto_supplemented_pedestrians.parquet.dvc
99+
# Person supplements (occupants/pedestrians) are NOT rebuilt in CI:
100+
# their deps are per-year `persons.parquet` side-effects of
101+
# `normalize.py`, which only runs locally after a manual `Crash.csv`
102+
# download. CI pulls these from S3 above (`dvx pull`) and feeds
103+
# them to the VTC enrichment + cmymc stages.
97104
- name: Rebuild cmymc.db (year-stats, victim-severity)
98105
if: ${{ !inputs.targets }}
99106
run: $DVX www/public/njdot/cmymc.db.dvc

0 commit comments

Comments
 (0)