Add algorithms docs for busline-usage-anomaly and service-by-operator

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: NoamGaash

algorithms/busline-usage-anomaly.md

@@ -0,0 +1,141 @@
+# Bus line usage anomaly — over- and under-used lines vs. their peers
+
+**Author:** team — ported from `busline_usage_anomaly.ipynb`, vendored at [busline_usage_anomaly.ipynb](../busline_usage_anomaly.ipynb) in the repo root
+**Code:** [analyses/busline_usage_anomaly.py](../analyses/busline_usage_anomaly.py)
+**Card:** `busline-usage-anomaly`
+**Data:** **data.gov.il** ticketing/validation open data — resource `ef42a264-9da2-41ad-9120-822064fb5433`. The only card here not built on Stride SIRI/GTFS.
+
+## What it answers
+
+Which bus lines carry unusually many or few passengers **for their peer group, at
+that hour of day**?
+
+The idea worth keeping: raw passenger counts cannot be compared across lines. A
+dense-city line and a suburban one carry wildly different volumes for reasons that
+have nothing to do with how well either is running. So lines are compared only
+against *peers*, and only within the same hour. A low score then means "carries
+fewer riders than comparable lines at the same time of day" — **not** merely "is a
+small line".
+
+## Algorithm
+
+1. **Fetch** `sample_rows` (default 15,000) from the data.gov.il datastore API,
+   paged 5,000 at a time.
+2. **Riders per row** — each row is one line × direction × hour × **month**, with
+   `D1..D31` daily counts. `riders = mean(D1..D31)`, nulls skipped (days the line
+   did not run, or a short month).
+3. **Clean** — drop the rail sentinel `OfficeLineId == -1` (rail rows share the
+   dataset and dwarf buses in volume) and drop `cluster_nm == "לא מוגדר"`.
+4. **Collapse** direction and month duplicates: `groupby(line, operator, cluster,
+   hour).riders.mean()`.
+5. **Peer z-score** — group by `(cluster, hour)`, compute `peer_mean`, `peer_std`,
+   `peer_count`, then `z = (riders - peer_mean) / peer_std`.
+6. **Filter** to `peer_count >= min_peers` (default 3).
+7. **Report** each line's single most extreme hour by `|z|`, top 14, sorted by z.
+
+## Reasoning
+
+Two deliberate departures from the notebook, **both because the original could not
+work as written** — this is the most substantive porting critique in the repo:
+
+**1. Peer grouping was rebuilt.** The notebook derived a "metro score" (exponential
+decay from the nearest of Tel Aviv / Jerusalem / Haifa) from station coordinates —
+but read it from data.gov.il resource `3ad014c3` (station passengers), **which has
+no line column at all**. Its `get_station_passengers(office_line_id)` silently
+ignored its argument and refetched the same global station table for every line, so
+**every line ended up scored off identical rows**. The port uses the ministry's own
+`cluster_nm` ("אשכול") — a real geographic/service grouping shipped in the same
+per-line dataset, which is what the metro score was proxying for anyway, and needs
+no join.
+
+**2. Hours were simplified.** The notebook mapped 7 coarse Hebrew time bands to an
+hour. The per-line resource carries a true `hour_a` (0–23), so no mapping is needed
+and the resolution is better.
+
+**Why `inputs=[]`.** Ticketing data keys on the ministry's own line ids, which do not
+line up with the SIRI/GTFS line + operator + date pickers in the global filter bar.
+Rather than accept filters it would silently ignore, the card declares it takes none.
+Given [route-divergence's finding 1](route-divergence.md) — a filter accepted and
+ignored producing a plausible wrong answer — this is exactly the right instinct.
+
+**Why `peer_std` gets an epsilon.** A peer group of one has no spread; replacing 0
+with `1e-5` keeps the division finite and lands its z at ~0, the honest reading for
+"nothing to compare".
+
+## Findings
+
+### 1. data.gov.il resource `3ad014c3` has no line column, and the notebook's per-line metro score was therefore meaningless — **confidence: High**
+
+`get_station_passengers(office_line_id)` ignored its argument and refetched the same
+global table for every line. Every line was scored off identical rows, so the
+resulting "metro score" varied not at all between lines. A schema fact plus a code
+path, both checkable; it is why the peer grouping was replaced rather than ported.
+
+**This is a finding about an analysis, not about buses** — recorded here because the
+whole point of these documents is that dead ends should not be rediscovered.
+
+### 2. `cluster_nm` is a usable ministry-defined peer grouping — **confidence: High**
+
+Shipped in the same per-line dataset, requires no join, and is the ministry's own
+geographic/service grouping. Directly observed in the data.
+
+### 3. The per-line resource carries a true `hour_a` (0–23) — **confidence: High**
+
+Better resolution than the 7 Hebrew time bands the notebook mapped from. Schema fact.
+
+### 4. Ticketing counts undercount anyone not validating — **confidence: High**
+
+Stated in the card's notes. Structural to validation data everywhere, not specific to
+Israel; the size of the undercount is unknown and unmeasured here.
+
+### 5. Which specific lines are over- or under-used vs. peers — **confidence: Low**
+
+The card's actual output. Three compounding reasons:
+
+- It is a **sample** (15,000 rows by default) of a much larger dataset, and the
+  sample is whatever the datastore returns first — **not random**. Which lines even
+  appear is an artifact of dataset ordering.
+- Peer groups need only 3 members by default. A z-score against 2 other lines has
+  almost no distributional meaning.
+- `riders` is a mean over a month's daily counts, so a line that ran 4 days scores
+  the same way as one that ran 30.
+
+The *method* is sound; the numbers it currently produces are not quotable.
+
+### 6. Counts of under- and over-performing line-hours (`z ≤ -1.5`, `z ≥ +1.5`) — **confidence: Low**
+
+Reported in the notes. With peer groups as small as 3, |z| ≥ 1.5 is not a rare event
+under any null hypothesis, and no multiple-comparison correction is applied across
+thousands of line-hours.
+
+## Criticism
+
+**"Sample the first 15,000 rows" is not sampling.** `offset` walks from 0 in dataset
+order, so the card analyses whichever lines the resource happens to list first.
+Everything downstream — which clusters have enough peers, which lines appear as
+extremes — inherits that ordering. A random sample, or full pagination with a longer
+cache TTL, would fix this and is the single highest-value change to the card.
+
+**Z-scores assume roughly normal peer distributions.** Ridership within a cluster is
+strongly right-skewed (a few trunk lines, many feeders), so z overstates how unusual
+the top end is and compresses the bottom. A rank-based or log-transformed score would
+suit the distribution better. The card's own framing — "not merely a small line" — is
+what makes this matter: skew is precisely the effect the z-score was meant to remove.
+
+**Peer groups mix operators and route types.** `cluster_nm` is geographic. Within one
+cluster, an express line and a local circulator are peers, and one of them will
+always look anomalous. The notes do not mention this.
+
+**The `min_peers = 3` default is far too low** for the statistic being computed, and
+the option's own help text ("a z-score against one other line is noise") suggests the
+author knew where the line was and drew it one step too permissively. With
+`peer_count = 3`, `peer_std` is estimated from three points.
+
+**Direction and month are averaged away** at step 4. A line that is packed inbound at
+08:00 and empty outbound at 08:00 shows as average. That is a real signal — arguably
+*the* signal for identifying under-served corridors — and it is discarded before
+scoring.
+
+**No date scoping at all.** The dataset spans multiple months; the card pools them
+with no window and no indication of which months are in play. Two runs with different
+`sample_rows` cover different time periods and are not comparable.

algorithms/service-by-operator.md

@@ -0,0 +1,112 @@
+# Planned vs actual rides — the worked example
+
+**Author:** example (repo scaffolding, by Noam Gaash)
+**Code:** [analyses/example_service_by_operator.py](../analyses/example_service_by_operator.py)
+**Card:** `service-by-operator`
+**Data:** Stride `/gtfs_rides_agg/group_by` + `/gtfs_agencies/list`
+
+## What it answers
+
+Daily count of rides planned (GTFS) against how many were actually observed (SIRI),
+for the selected operators. **The gap is unrun service.**
+
+This file exists to be copied — it is the template a hackathon participant reads
+before writing their own analysis. It hits the real API, handles the empty case,
+returns two different result kinds, and is deliberately short.
+
+It has therefore ended up in an awkward position: **it is the clearest statement of
+the question the whole hackathon is about, built on the one endpoint that cannot
+answer it.**
+
+## Algorithm
+
+1. `stride.gtfs_rides_agg(date_from, date_to, group_by="operator_ref,gtfs_route_date")`
+   — pre-aggregated server-side, much kinder than paging every individual ride.
+2. Merge `stride.agencies()` for human-readable names; fall back to the numeric
+   `operator_ref` where the name is missing.
+3. Filter to `req.operators` if any are selected.
+4. Resolve the planned and actual columns **by trying several names** —
+   `_first_col(agg, ["total_planned_rides", "planned_rides", "num_planned_rides"])`
+   and the same for actual.
+5. Sum per day, melt to long form, render as a two-series line chart.
+
+## Reasoning
+
+**Why `gtfs_rides_agg` at all.** For a template, the pedagogically right move is to
+show the cheap, pre-aggregated, server-side path rather than teach newcomers to page
+raw rides against a shared community API. The instinct is correct and the guidance
+in [CLAUDE.md](../CLAUDE.md) repeats it.
+
+**Why `_first_col` tolerates renames.** *"Tolerate small API column renames instead
+of hard-failing mid-demo."* The whole repo is built around one broken analysis not
+taking the demo down, and this is that principle at the column level.
+
+**Why it degrades to planned-only.** If no actual column is found it notes
+*"No actual-rides column returned by /gtfs_rides_agg — showing planned only"* rather
+than charting a misleading zero.
+
+**Why `inputs=["operators", "dates"]` and not lines.** It aggregates across whatever
+operators are selected, so a line filter would be misleading.
+
+## Findings
+
+### 1. The aggregate's actual-rides column is present but always zero — **confidence: High**
+
+This card's own graceful degradation does not trigger, because the problem is not a
+missing column. `total_actual_rides` **exists and is populated with 0** for every
+row, network-wide, on every date sampled Nov 2025 → Jul 2026. See
+[days-with-no-cancellations](days-with-no-cancellations.md) finding 1 for the
+measurements and the control that rules out ingestion lag, and
+[service-violations](service-violations.md) finding 2 for the independent
+confirmation.
+
+**The consequence for this card specifically:** the note it prints reads
+*"Overall 0.0% of planned rides were observed"* and the chart draws a flat line at
+zero beneath the planned series. To a viewer, that is a total collapse of Israeli bus
+service. It is a broken column.
+
+### 2. Graceful degradation designed against renames does not protect against wrong values — **confidence: High**
+
+A design lesson worth stating plainly. `_first_col` guards the schema; nothing guards
+the semantics. The card is *most* wrong precisely where it looks most confident,
+because the defensive path it was given never fires.
+
+### 3. Planned-ride counts from the aggregate are trustworthy — **confidence: Medium**
+
+`num_planned_rides` is populated and, per orion's cross-check, **agrees closely with
+the ride-level endpoint** (line 2259 / 2026-07-29: aggregate 123 planned vs 126 from
+`/rides_execution/list`). That write-up also carries an explicit correction of an
+earlier claim that the two disagreed. Medium: one line, one date, and 123 vs 126 is
+close but not equal.
+
+So the endpoint is usable **as a planned-ride denominator** — which is exactly the
+recommendation orion's method doc lands on.
+
+## Criticism
+
+**As shipped, this card tells the audience that no bus in Israel ran.** It is the
+example card, so it is the one most likely to be looked at first, and it has no note
+warning that the actual series is a known-broken column. Every other card that
+touched this endpoint documented the defect at length in its notes; the template did
+not. Given the hackathon's presentation is the deliverable, that is the most
+consequential single fix in the repo:
+
+- add a note when `actual_total == 0` and `planned_total > 0` saying the aggregate's
+  actual column is unpopulated on this deployment, linking the evidence; **or**
+- drop the actual series and retitle the card to what the endpoint can honestly
+  deliver — planned service volume per operator per day.
+
+**It teaches the wrong first step.** A newcomer copying this file learns to reach for
+`gtfs_rides_agg` for planned-vs-actual, which is the one thing it cannot do here.
+`/rides_execution/list` (see [days-with-no-cancellations](days-with-no-cancellations.md))
+is the endpoint that works, and the template does not point at it.
+
+**No SIRI lag clamp.** Unlike essentially every other card, this one uses
+`req.date_from`/`req.date_to` directly. With a healthy actual column it would show a
+sharp fake decline over the last 3 days — a second way to read a data artifact as a
+service collapse.
+
+**`daily["Actual"] = agg.groupby(...).sum().to_numpy()`** assigns by position, not by
+join key. It happens to be safe because both aggregations group the same frame by the
+same column in the same sorted order, but it is a positional assignment in the file
+that exists to be copied.