feat(bus): Keep original values for gtfs_arrival_dt and gtfs_departure_dt - #795
feat(bus): Keep original values for gtfs_arrival_dt and gtfs_departure_dt#795runkelcorey wants to merge 8 commits into
gtfs_arrival_dt and gtfs_departure_dt#795Conversation
LCOV of commit
|
| ) # use the first in transit dt from the previous stop) | ||
| .when(pl.col("point_type").eq(pl.lit("end"))) # endpoints | ||
| .then(pl.lit(None)) # no departure time | ||
| .otherwise( # midpoints |
There was a problem hiding this comment.
Hi @runkelcorey, I hope that orientation went well! We tried to do an assessment of the potential changes by comparing staging to prod, and I think a middle ground would be the most useful thing for us to have to test, but we aren't ready to move this proposal into prod yet.
When you're back, can you please put the following changes into staging for midpoints (lines 225 to 232), so that the stop_departure_dt is the result of coalescing:
- min_horizontal(
tm_actual_departure_dt,gtfs_departure_dt) - the next
gtfs_first_in_transit_dt(exactly what's in lines 230 to 232, but this is what would be coalesced, instead of including this in the previous step where you take the earliest of the various timestamps.
We did some analysis we can talk more about next week, and in the cases where there actually IS a tm_actual_departure or a gtfs_departure_dt, taking the next stop's gtfs_first_in_transit_dt reduced accuracy, so we want to test out just having this be a fallback when the data is null, instead of treating it as one of the equally viable options in the min_horizontal step.
gtfs_last_in_transit_dtNOTgtfs_first_in_transit_dt(the final option for the coalesce in line 234)
The final fallback for the stop departure should be the last in_transit_to record approaching the stop instead of the first recorded timestamp approaching the stop
Let me know if you want to huddle about any of these proposals before implementing!
Working group needs to be able to reconstruct the
stop_arrival_dtandstop_departure_dtfrom its components.What changes does this PR propose?
Refactors to defer creating
stop_arrival_dtandstop_departure_dtuntil the last stage.How were these changes validated?
What questions should reviewers consider?
None.