Document BusAnalysis related work and start algorithms/ solution docs

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: NoamGaash

CLAUDE.md

@@ -86,7 +86,12 @@ not the shared community API. `gtfs_rides_agg`'s `group_by` only accepts the
 fields in `AGG_GROUP_BY_FIELDS` (`gtfs_route_date`, `gtfs_route_hour`,
 `operator_ref`, `day_of_week`, `line_ref`) — anything else 500s server-side,
 so it's validated client-side first. Prefer `gtfs_rides_agg`/`siri_rides`
-over paging raw endpoints when the shape fits.
+over paging raw endpoints when the shape fits. Known defects in the upstream
+data — the stored SIRI→GTFS ride link is empty since 2024-10, `siri_ride` has
+~2.6% duplicate journeys, `first_vehicle_location_id` is a processing-state
+flag rather than a transmission signal, and five operators never reach the
+feed at all — are documented in `docs/busanalysis.md`. Check it before
+building an analysis that joins planned to actual or reads those columns.
 
 **Styling is shared between Python and TypeScript by convention, not by
 import**: `openbus_hack/theme.py`'s `SERIES_LIGHT`/`SERIES_DARK` palettes are

README.md

@@ -158,6 +158,15 @@ disk and won't hammer the shared DB.
 | Line info lookup (e.g. line 86001) | https://markav.net/line/86001/ |
 | Hasadna Slack (#opendata / open-bus channels) | https://join.slack.com/t/hasadna/shared_invite/zt-458cp0v0n-GSnHzAq6F5aHeK43O4YeeA |
 
+**Related work — read before trusting a number:**
+[`docs/busanalysis.md`](docs/busanalysis.md) summarises
+[lihay7/BusAnalysis](https://github.com/lihay7/BusAnalysis) (private), a national
+planned-vs-actual reconstruction on this same data. It documents **five defects in
+the upstream `stride` tables** that constrain what our analyses can honestly claim —
+most importantly that the stored SIRI→GTFS ride link has been **empty since October
+2024**, and that `first_vehicle_location_id` is a processing-state flag, not evidence
+a vehicle stopped transmitting.
+
 ---
 
 ## Your scratch space

algorithms/README.md

@@ -0,0 +1,85 @@
+# Algorithms — what each hackathon solution actually does
+
+One file per solution. Each documents the **author**, the **algorithm**, the
+**reasoning** behind the design choices, the **findings** (each with a stated
+confidence level), and **criticism** — where the output should not be trusted,
+and what would have to change for it to be.
+
+These are working notes for two audiences: whoever picks a POC up for
+consolidation upstream, and whoever has to defend a number in the
+presentation. Neither is served by a document that only says what worked.
+
+## Index
+
+| Solution | Author | Cards | What it answers |
+|---|---|---|---|
+| [bus-arrival-reliability.md](bus-arrival-reliability.md) | noamf2001 | 3 | Where is the timetable optimistic? Where does the bus lose time? Which segments break at rush hour? |
+| [schedule-adherence.md](schedule-adherence.md) | yuvalko1 | 3 | How much does the same departure vary day to day, geographically and per day? |
+| [siri-coverage.md](siri-coverage.md) | yuvalko1 | 1 | What share of planned stops actually got a real-time GPS ping, by hour? |
+| [gps-trace-map.md](gps-trace-map.md) | yuvalko1 | 1 | What does one real ride's raw GPS trail look like? |
+| [poisson-arrival-regularity.md](poisson-arrival-regularity.md) | yuvalko1 / Yuval | 1 | Does headway spacing decay into a random Poisson process downstream? **Currently broken.** |
+| [days-with-no-cancellations.md](days-with-no-cancellations.md) | orion | 1 (+CLI) | What fraction of the last 15 days did a line run with zero cancellations? |
+| [service-violations.md](service-violations.md) | team | 2 | Ghost rides, early departures, late departures — the three fineable failure modes. |
+| [bus-bunching.md](bus-bunching.md) | team | 1 | How evenly spaced were consecutive buses against the line's own scheduled spacing? |
+| [route-divergence.md](route-divergence.md) | team | 2 | Which buses strayed from the planned route, and where? |
+| [busline-usage-anomaly.md](busline-usage-anomaly.md) | team | 1 | Which lines carry unusually many/few riders for their peer group and hour? |
+| [service-by-operator.md](service-by-operator.md) | example | 1 | Planned vs actual rides per day — the worked example. |
+
+Related, not a hackathon solution: [`docs/busanalysis.md`](../docs/busanalysis.md)
+summarises `lihay7/BusAnalysis`, an independent national reconstruction on the
+same data whose five upstream defect reports overlap heavily with what these
+cards hit.
+
+## Upstream issues
+
+[upstream-issues.md](upstream-issues.md) collects every defect these solutions
+ran into, deduplicated and written as paste-ready issue drafts, with the repo
+each one belongs in. That is the prep for filing against
+[hasadna/open-bus-map-search](https://github.com/hasadna/open-bus-map-search)
+and its sibling repos.
+
+## Confidence levels
+
+Every finding below carries one of these. They describe **how much the finding
+would survive scrutiny**, not how large the effect is.
+
+| Level | Means |
+|---|---|
+| **High** | Directly observed in the data, on a population large enough that sampling isn't the explanation, and the mechanism is understood. Would survive being quoted in public. |
+| **Medium** | Observed and reproducible, but on one line / a few days / one operator, or resting on a proxy whose error is understood but not bounded. Directionally trustworthy; the exact number is not. |
+| **Low** | Observed once, or on a sample too small to separate from noise, or resting on a proxy that could be an artifact. Worth investigating, not worth quoting. |
+| **Not a finding** | An artifact of the data pipeline that was mistaken for a finding at some point during the hackathon, and is recorded here so nobody rediscovers it as real. |
+
+A structural rule that applies to nearly everything here: **these are claims
+about the record, not about the road.** A ride SIRI never saw is
+indistinguishable from a ride that never ran. Where a card counts "missing"
+rides, the finding is about what the tracking feed contains.
+
+## Shared method, shared caveats
+
+Most cards converge on the same skeleton, for the same reasons — worth reading
+once rather than in every file:
+
+- **Line resolution.** A "line" (`route_short_name` like `480`) is *not* one
+  thing. It is many `line_ref`s — one per direction × route alternative ×
+  operator — and the set changes over time (line 480 was 2 refs in Nov 2025 and
+  8 in Jul 2026). Cards resolve `route_short_name → line_ref` through
+  `/gtfs_routes/list` **for the window being analysed**, and most take the first
+  match, which is a real simplification.
+- **SIRI lag.** Ingestion runs ~3 days behind live. Every card clamps its window
+  back by `LAG_DAYS = 3`; without it, a perfectly healthy line returns zero
+  pings and reads as a total service collapse.
+- **Weekend exclusion.** Israeli bus service is thin by design on Fri/Sat, so
+  headway- and reliability-based cards drop `weekday() in (4, 5)`.
+- **Duplicate pings.** Overlapping SIRI snapshots repeat the same physical
+  observation — measured at ~10% of rows. Every card that touches raw pings
+  dedups on `(siri_ride__id, recorded_at_time, lat, lon)`.
+- **First-ping-as-departure.** No card observes a real "doors closed" moment.
+  Actual departure is inferred from GPS, and the naive version of that inference
+  is wrong in a specific, measured way — see
+  [service-violations.md](service-violations.md), which is where the artifact
+  was found and corrected.
+- **Cost caps.** These run as live dashboard cards, not batch jobs. Where an
+  original notebook scanned 90 days, the port scans 2–21 and says so in its
+  notes. Lower confidence is the price, and it is disclosed on the card rather
+  than hidden.

docs/busanalysis.md

@@ -0,0 +1,122 @@
+# BusAnalysis — related work on the same data
+
+[`lihay7/BusAnalysis`](https://github.com/lihay7/BusAnalysis) is an independent
+reconstruction of Israel's planned-vs-actual bus record, built on the same
+hasadna `stride` data this hackathon uses. It is worth knowing about here for
+two reasons: it answers "did the scheduled bus run?" at national scale, and it
+found five defects in the upstream data that affect anyone querying the same
+tables — including our analyses.
+
+**Access:** the repo is **private** (a plain `curl` or unauthenticated fetch
+404s — that is permission, not a missing repo). Licence is MIT, but it is not
+our code: anything ported across needs `lihay7`'s sign-off first. Read it with
+an authenticated `gh`:
+
+```bash
+gh repo view lihay7/BusAnalysis
+gh api repos/lihay7/BusAnalysis/git/trees/main?recursive=1 --jq '.tree[].path'
+```
+
+Scope: 1,306 service days (2023-01 → 2026-07), 117.2M schedule links, ~7 GB
+local warehouse rebuilt from GTFS + SIRI. Created and pushed 2026-07-31.
+Languages are Python (~1 MB of source) and HTML (the committed dashboards).
+
+---
+
+## The four findings
+
+| # | Finding | Fix direction |
+|---|---|---|
+| 1 | **5.2%** of scheduled departures never appear in the national tracking record (±5 min, 116.4M rides, tracked operators). Widening to ±60 min only moves it to 4.9% — so these are cancellations, not delays. | Publish the rate per line and per month **with its window stated**. |
+| 2 | **~1.7×** — the sparsest lines (under 8 departures/day) lose **7.8%** of departures against **4.6–4.8%** on mid-frequency lines, after the control that kills the naive version of the claim. | Protect thin lines first: one missing bus on a two-a-day line is the whole service that day. |
+| 3 | **5.0% vs 1.5%** — reconstructed non-execution for 2024-H1 against the figure the ministry's own electronic control published for the same period, both at ±30 min. | Compute the violation rate from the open record; the enforcement basis is already electronic. |
+| 4 | **4.2 minutes** median *extra* waiting beyond what the published timetable promises, nationally, every month. 11.8% of line-months run 10+ minutes over promise. | Publish a timetable the service can keep — an honest every-20 beats a broken every-10. |
+
+Each number is reproducible from committed aggregates in `report/metrics/`
+(parquet, ~9 MB) without rebuilding the warehouse.
+
+---
+
+## Five upstream data defects — relevant to our analyses
+
+Written up as paste-ready issues in `handoff/issues/`. These describe the
+`stride` database we query through [stride.py](../openbus_hack/stride.py), so
+they constrain what our own cards can honestly claim.
+
+| ID | Defect | Why it matters to us |
+|---|---|---|
+| **F1** | The SIRI→GTFS ride-matching job has written **zero matches since October 2024**. All four match columns (`gtfs_ride_id`, `route_gtfs_ride_id`, `scheduled_time_gtfs_ride_id`, `journey_gtfs_ride_id`) fail together; the raw feed is healthy (~2.9–3.1M `siri_ride` rows/month). | Any analysis joining actual→planned via the stored link gets nothing for the last ~21 months. You have to match yourself. |
+| **F6** | **2.6%** of `siri_ride` rows are surplus duplicate journeys (3.93% sit in a duplicated group); rate has ~quadrupled since 2023 and is 66× skewed across operators. Downstream of the `scheduled_start_time` drift bug (hasadna #390). | Raw counts of tracking rows overstate real service. Affects anything counting rides rather than deduplicating journeys. |
+| **F7** | `first_vehicle_location_id` is **100% null for 18 consecutive months** (2024-12 → 2026-05, 49.4M rides); `duration_minutes` tracks it. | It is a **processing-state flag, not a property of the bus** — it encodes whether stride's enrichment job ran, not whether a vehicle transmitted. Easy to misread as a per-operator transmission metric. Don't. |
+| **F8** | The GTFS import sometimes keeps the previous release alongside the current one under a single `gtfs_route.date`, **doubling planned counts** on affected dates. 50 days exceed 1.5×, 36 exceed 1.8×, scattered with no era pattern. | Planned-ride denominators are ~2× too high on those dates. |
+| **F9** | **Five operators never reach the tracking feed** — three absent entirely across 3.5 years, two under 1% covered. 2.74M scheduled rides, ~2.3% of national planned volume. | They are unmeasurable, not failing. Counting them as cancellations is most of how 5.2% becomes 7.4%. |
+
+Fixing all five moves the national figure from **7.4% → 5.2%** (and to 42% on
+the worst days), which is the case for treating them as defects rather than
+noise.
+
+---
+
+## How to read numbers off this data honestly
+
+Four rules the project holds itself to. They apply to our cards too:
+
+1. **Every rate names its time window.** There is no tolerance-free answer to
+   "did the bus run?" A figure without a window can't be compared to one that
+   has it.
+2. **These are claims about the tracking record, not about the road.** A ride
+   the feed never saw counts as missing. No independent national bound exists
+   on the feed's own completeness — so the phrasing is "never appears in the
+   record", never "the bus didn't run".
+3. **Time differences here are scheduled-against-scheduled, not punctuality.**
+   The record holds no observed departure time. Their M2 is a *waiting*
+   measure; actual lateness needs stop-level data.
+4. **Five operators have no tracking feed at all** — excluded from every
+   performance rate and reported as a finding in their own right (F9).
+
+---
+
+## How it relates to this repo
+
+Same data, opposite architecture:
+
+| | this repo | BusAnalysis |
+|---|---|---|
+| Data access | live paginating API client with a disk cache | ~7 GB local mirror (4 slim tables, 245M rows), not committed |
+| Unit of work | one `@analysis` function per person → card | a batch pipeline: mirror → per-day matcher → marts → metrics → figures |
+| Matching | not attempted; we lean on what the API returns | own greedy 1:1 matcher, 6 time windows, in `src/busanalysis/matching/greedy.py` |
+| Output | live React dashboard over FastAPI | committed self-contained HTML (`report/dashboard/index.html`, 11.6 MB; `brief.html`, one screen) |
+| Registry | `openbus_hack/registry.py` discovers `analyses/*` | has its own `src/busanalysis/registry.py` for the same job |
+
+Layout of theirs, briefly:
+
+```
+src/busanalysis/
+  pipeline/    mirror, extract_day, marts, backfill
+  matching/    greedy 1:1 matcher + validation
+  quality/     filters
+  metrics/     departure_fidelity, enforcement_gap, gap_series,
+               record_integrity, city_profiles
+  viz/         charts, dashboard, brief, city_map, heatmap
+tests/         substantial — test_extract_day.py alone is 42 KB
+report/        dashboards, figures, committed parquet aggregates
+handoff/       the five defect issues + QUESTIONS_FOR_HASADNA.md
+assistant/     research record: findings ledger, verification memos,
+               DECISIONS.md, ATTEMPTS_AND_FAILURES.md (the dead ends)
+plans/         the plans each phase was executed from
+```
+
+## Directions worth taking from it
+
+- **Port finding #2 as an analysis card.** "Sparse lines lose disproportionately
+  more service" maps cleanly onto `@analysis` + `bar_chart` over
+  `gtfs_rides_agg`, and it is the finding that most needs a live, per-line view
+  rather than a static national figure.
+- **Cross-check.** They built from an offline mirror, we query the API live. If
+  our numbers diverge from their 5.2%, the divergence itself is informative —
+  most likely F6/F8 territory.
+- **Carry the defect list upstream.** The five `handoff/issues/` write-ups are
+  directly relevant to the consolidation this hackathon feeds into
+  (`open-bus-stride-api`, `open-bus-pipelines`). F1 in particular claims the
+  national "rides that didn't run" statistic has had no computable basis since
+  October 2024.