fix: Remove exclude-hours from some API calls

[arielvino]

Description

Removes the hardcoded exclude_hour_from=23&exclude_hour_to=2 that the dashboard has sent to gtfs_rides_agg/group_by since 2023 (#6, commit 891bc0e).

Why remove it:

1. Unclear why it's there.
   It was added to hide "edge cases in the EOD hours" (Added gtfs_rides_agg_by_hour View migration open-bus-stride-db#21, gtfs_ride_agg API adaptations to the new view open-bus-stride-api#27)
   but those edge cases were never documented anywhere.
2. The data is fine now.
   I verified directly against the production API: the night/early-morning hours look completely normal (today and as far back as
   the data goes, including the week the exclusion was introduced). Whatever happened in 2023 was likely a transient issue.
3. The exclusion itself is buggy.
   The aggregation view bins hours in UTC while the frontend means Israel time, so "exclude 23:00–02:59" actually drops 02:00–05:59 Israel time - silently hiding the 05:00 morning ramp-up, ~3% of all planned rides - while the real midnight hours stay in. (Same
   UTC-vs-Israel bug class as Serious bug: rides_execution/list uses UTC midnight for date filtering instead of Israel midnight. Possibly the root cause of multiple other issues. open-bus-stride-api#54; will be reported separately against stride-api/stride-db along with the verification queries.)

Expected effect: dashboard/operator totals rise by ~3% - previously hidden early-morning data coming back, not a regression.
Anyway, since operator/dashboard aggregation is currently broken at the ETL level - this will not affect the frontend till those be fixed - see
hasadna/open-bus-stride-etl#22.

[github-actions]

Preview: https://s3.amazonaws.com/noam-gaash.co.il/27346901537/open-bus/2416/index.html
Preview Storybook: https://s3.amazonaws.com/noam-gaash.co.il/27346901537/open-bus/2416/storybook/index.html
Test Report (if available): 'https://s3.amazonaws.com/noam-gaash.co.il/27346901537/open-bus/4272cd5e2d21973645190b038ad032c07b93cfac/test-results/index.html'

[arielvino]

@NoamGaash @OriHoch @ShayAdler
Do you remember what was going on back then, or can confirm whether it needed anymore or not?

[NoamGaash]

@arielvino when you're saying that the data is fine, what do you mean by that? how did you test it?
https://open-bus-map-search.hasadna.org.il/he/dashboard

the reason we ignored the midnight times is because we had a theory that maybe the service-day duration model isn't aligned with what we implemented in our DB, and maybe some plans doesn't have a matched ride execution because of timezone, daylight saving time and service hours related issues

[arielvino]

The frontend dashboard show nothing since ~October 2024 because an ETL edit caused the aggregation in the database to stop proccessing new data.

However, the rides data itself is still intact and can be fetched and compared via other API endpoints.
I did some samples of rides counts from different dates - and saw that the rides count by hour looks correct and that the current exclude as the frontend use it is buggy - but I didnt checked individual rides at those edge cases.

Now I tasked claude to sample them. The following is claude anlysis - unverified yet.

(Anyway since the frontend show nothing currently - I will abandon this till it fixed.
Operator page maybe able to walkaround using other API calls and frontend matching, but dashboard need ETL and aggregation fixes because of performance demand)

Individual-ride verification: does midnight/DST break plan↔execution matching?

Tests the original theory behind the dashboard's exclude_hours ("maybe some plans don't have a matched ride execution because of timezone, daylight
saving time and service hours related issues") — at the level of individual rides and their timestamps, not aggregate counts.

Method

A single endpoint exposes everything needed: siri_rides/list?expand_related_data=true returns, per SIRI ride, its matched GTFS ride's planned time
(gtfs_ride__start_time), the plan's service date (gtfs_route__date), the journey_ref (whose prefix is the service date), and monitoring evidence
(first_vehicle_location_id). For each ride we check:

1. delta = scheduled_start_time − gtfs_ride__start_time (a timezone/DST bug would show as ±1h/±2h/±24h offsets)
2. day attribution: gtfs_route__date vs the Israel-calendar day of the scheduled time (Asia/Jerusalem, real DST rules)
3. journey_ref date prefix vs the same
4. matched at all: gtfs_ride_id != null

Fetch a window (datetime filters are timezone-aware; + must be URL-encoded as %2B):

curl -s 'https://open-bus-stride-api.hasadna.org.il/siri_rides/list?limit=300&order_by=scheduled_start_time%20asc&scheduled_start_time_from=2024-06-03T23:3
0:00%2B03:00&scheduled_start_time_to=2024-06-04T00:30:00%2B03:00&expand_related_data=true' -o midnight.json

Analyze per ride:

import json, collections
from datetime import datetime
from zoneinfo import ZoneInfo
IL = ZoneInfo('Asia/Jerusalem')
rides = json.load(open('midnight.json'))
deltas, day_mm, jref_mm, unmatched = collections.Counter(), 0, 0, 0
for r in rides:
    sched = datetime.fromisoformat(r['scheduled_start_time'])
    il_day = sched.astimezone(IL).date().isoformat()
    if r['journey_ref'][:10] != il_day: jref_mm += 1
    if r['gtfs_ride_id'] is None: unmatched += 1; continue
    deltas[(sched - datetime.fromisoformat(r['gtfs_ride__start_time'])).total_seconds()] += 1
    if r['gtfs_route__date'][:10] != il_day: day_mm += 1
print(len(rides), 'rides | unmatched:', unmatched, '| deltas:', dict(deltas),
      '| day mismatches:', day_mm, '| journey_ref mismatches:', jref_mm)

A fast unmatched-rate probe for any date (used for the sweeps below; swap the date in both params):

curl -s 'https://open-bus-stride-api.hasadna.org.il/siri_rides/list?limit=300&scheduled_start_time_from=2024-03-28T10:00:00%2B00:00&scheduled_start_time_to
=2024-03-28T10:20:00%2B00:00' | jq '[length, ([.[] | select(.gtfs_ride_id == null)] | length)]'

Result 1 — regular midnight crossing: clean

Window 2024-06-03 23:30 → 06-04 00:30 Israel time, 300 rides:

check	result
matched	289/300 (96.3%)
delta scheduled vs planned	0 seconds for all 289 — exact to the second
day attribution (incl. 00:0x → next day, 23:5x → same day)	300/300 correct
journey_ref date	300/300 correct
monitored (has vehicle locations)	300/300

Daytime control (2024-06-04 10:00 IL, 1500 rides): 2.8% unmatched, delta 0 for 1447, a small tail of ±1–4 min (matcher tolerance). So the midnight window
is indistinguishable from daytime. The midnight theory is false at the individual-ride level too.

Result 2 — DST spring-forward (Fri 2024-03-29, 02:00→03:00): real, but a whole-day outage

Unmatched rate at noon (UTC window, same probe command per date):

date	unmatched
2024-03-25/26/27	0/300, 0/300, 3/300
2024-03-28 (eve of switch)	300/300 — and 1481/1500 in the 23:00 window
2024-03-29 (switch day)	1496/1500 at noon, 1498/1500 in the 03:00–06:00 morning
2024-03-30 / 03-31 / 04-01	12/191, 0/300, 4/300

Planned GTFS data exists for both broken days (116,805 and 60,741 rides — gtfs_rides/list?get_count=true&gtfs_route__date_from=...), so it's the
matching that collapsed, not the data. The few rides that did "match" are wrong: 19 rides at delta −47 h (matched to the 2024-03-30 plan at the same
wall-clock time — the missing DST hour visible in the offset), others at −24 h with next-day gtfs_route__date.

Result 3 — DST fall-back (Sun 2023-10-29, 02:00→01:00): essentially healthy, trace artifacts

Transition days: 5/300 and 4/300 unmatched at noon — baseline. In a 1,500-ride evening sample: delta 0 for 1,429, the usual ±1–4 min tail, and 2 rides
matched exactly −3600 s (1 h off) + 2 matched to next-day plans (~0.3%).

Result 4 — byproduct: the exact date the matcher died

2025/2026 DST transitions are untestable because siri_ride.gtfs_ride_id is 100% null for everything recent (the known dead enrichment ETL). Bisecting its
onset with the same probe: 2024-07-01 → 0/300, 2024-09-20 → 0/300, 2024-09-23 → 0/300, 2024-09-24 → 300/300, and 100% ever since. The breakage
predates the fall-2024 DST switch — unrelated to DST.

Conclusions

1. Midnight/service-day misalignment: disproven. Around midnight, matching is exact-to-the-second, day attribution is calendar-day-consistent on both
   sides, and the unmatched rate equals daytime.
2. The DST kernel of the theory is real — spring transitions break matching — but as ~2 fully-broken days per year, not as a night-hours
   phenomenon: the outage covers noon as much as midnight, and fall transitions barely register (~0.3%).
3. None of this ever justified the dashboard exclusion: matching failures affect only actual counts; num_planned_rides (what exclude_hours
   filters) is pure GTFS and untouched by any of it.
4. Follow-up for open-bus-stride-etl: investigate the matcher's date handling across spring DST transitions (the −47 h / −24 h mismatches point at a
   date-keyed join meeting the shifted local day), independently of restoring the dead enrichment chain (broken since 2024-09-24).