Rank operators by days-without-cancellations (the unbuilt "method 3")

[NoamGaash]

⚠️ AI-generated draft — needs human validation

This issue was written by an AI agent from materials produced during the
hasadna Open Bus hackathon (July 2026). The underlying analysis was built fast,
under hackathon conditions, and has not been peer-reviewed. Figures,
endpoint behaviour and conclusions all need independent verification before
anyone acts on them or quotes them publicly.

Please validate before implementing. Corrections very welcome.

🛑 On hold — do not start implementing

The metric this ticket proposes to build has a design problem that is not yet
resolved (see "Why this is on hold" below). The ticket is kept open because
the question it asks is worth answering; the method it proposes is probably
the wrong way to answer it. If you want to work on this, the useful contribution
right now is deciding what the metric should be — not writing the code.

Background — what this is about, from scratch

You do not need to have followed the hackathon to read this. Here is the whole story.

There is no "cancelled" flag in the Israeli bus data. A cancellation has to be
inferred. The Stride API has an endpoint, /rides_execution/list, that returns one
row per planned ride with two columns that matter: planned_start_time and
actual_start_time. If actual_start_time is null, that planned ride has no
recorded execution — which is usually a cancellation.

During the hackathon, someone built a reliability score on top of that. The idea:
for a given bus line, look at the last 15 service days. A day where every planned
ride ran counts as 1. A day with even one cancellation counts as 0. The score is
good_days / days_scored. It is deliberately harsh — the argument was that "did my
line have a clean day" is closer to lived passenger experience than "99.4% of trips
operated".

That work defined three methods, in a code comment
(orion/days_with_no_cancellations.py:6-11):

		Built?	Ticket
Method 1	score one line	✅ yes	#1795
Method 2	score every line of one operator, ranked worst-first	✅ yes	#1796
Method 3	compare operators against each other	❌ never built	this ticket

Method 3 is one sentence of a comment — # method 3: script that compares average score per bus company, and provide a visualization. There is no spec beyond that.
The obvious reading is: run method 2 for every operator, average each operator's
line scores, rank the operators.

That is what this ticket originally proposed. It should probably not be built as
described. Here is why.

Why this is on hold

@arielvino raised the objection that a
binary per-day metric is arbitrary and misleading:

A day without cancellation is a binary question. You don't know if the outage was
one single missing ride, or a 50% of all planned rides, for example. So it can flag
operators with constant small misses as worse than operators with frequent severe
outages, just because the first is smeared upon more calendar dates.

This was checked against the live API and it is correct. 52 lines across 9
operators, 15-day window (2026-08-01 → 2026-08-15), scoring logic copied from the
hackathon implementation, with a volume-weighted rate (1 − cancelled/planned)
computed from the same rows for comparison:

1. Two lines with near-identical real service, scored 6.7× apart:

line	rides/day	trips actually operated	binary score
תנופה 11	26	89.7%	0.47
בית שמש אקספרס 12	113	87.8%	0.07

The line that runs more of its trips scores worse, because it has more chances
per day to miss one.

2. Lines with an identical score delivering very different service. Five sampled
lines scored exactly 0.20. Their actual operated-trip rates ranged from 83.4%
(גלים 494) to 96.4% (סופרבוס 1) — a 13-point spread compressed into one number.
Two lines of the same operator, גלים 492 and 494, both score 0.20 while running
94.1% and 83.4% of their trips.

3. The score tracks frequency, not just reliability. Spearman correlation between
the binary score and rides/day is −0.69.

rides/day	lines	mean binary score	mean trips operated
< 20	17	0.90	99.2%
20–59	16	0.67	95.5%
60–119	10	0.41	95.6%
≥ 120	6	0.46	98.3%

A ~1 percentage-point difference in delivered service between the top and bottom
band shows up as a ~0.5 difference on a 0–1 headline score.

4. And that propagates straight into the operator ranking this ticket asks for.
Averaging each operator's line scores — literally what method 3 does — produced this:

operator	lines sampled	mean-of-binary (what method 3 publishes)	trips actually operated	median rides/day
מועצה אזורית גולן	6	0.96	99.6%	11
גי.בי.טורס	6	0.83	99.4%	34
תבל	2	0.77	99.8%	245
סופרבוס	10	0.72	97.1%	35
גלים	10	0.70	98.0%	47
תנופה	5	0.53	85.2%	26
בית שמש אקספרס	10	0.37	95.1%	96

תנופה delivers 10 percentage points less service than בית שמש אקספרס and ranks
above it. The ranking is inverted for the two worst operators in the sample,
because תנופה's average is propped up by two low-frequency lines that score 1.00 and
0.93. A national table built this way would partly be measuring which operators run
rural low-frequency routes, and publishing it as an accountability metric.

In fairness to the metric — it is not noise. Within a single frequency band it
discriminates cleanly: at ~230 rides/day, תבל lines 1 and 3 score 0.80/0.73 (99.8%
operated) while סופרבוס lines 1 and 71 score 0.20/0.27 (96.4% operated). Cancellations
also genuinely cluster on particular days — סופרבוס line 1 scores 0.20 where an
independent-failure model predicts ~0.00005, so "some days are bad days" is a real
effect the metric captures. The problem is specifically cross-line and
cross-operator comparison, which is the only thing method 3 does.

What would actually be worth building

Pick one of these and say so on the ticket before starting:

Option A — publish a volume-weighted execution rate instead (recommended).
Pooled 1 − Σcancelled / Σplanned per operator. It is computable from the exact same
API rows at zero extra cost, it is comparable across operators, and it is what most
transit regulators publish. This is what the hackathon author themselves recommended
in the write-up's own criticism section. The binary score can stay as a secondary
"clean days" column, clearly labelled as not comparable across lines.

Option B — keep the binary score but make it comparable. Band lines by frequency
and only ever compare within a band, or normalise against an expected clean-day rate
given the line's trip count. Harder, and it still throws away the severity
information @arielvino points out.

Option C — close this ticket. Methods 1 and 2 (#1795, #1796) are per-line
diagnostics and stand on their own merits. If nobody wants to defend a
cross-operator ranking, that is a legitimate outcome and worth recording.

Blocker: feed gaps are not cancellations, and the existing guard misses them

Whatever metric is chosen, this has to be solved first, because it can manufacture a
"worst operator in Israel" out of nothing.

The hackathon code guards against this with no_actuals = (cancelled == planned) —
if a line reports zero actuals for the entire window, it is drawn grey rather than
scored 0.00. That guard is not sufficient. Verified live in the same window:

• כרמלית (operator_ref 20) — 2,040 planned rides, 2,040 "cancelled". Confirmed to
  have zero rows in /siri_rides/list for the window. It is the Haifa funicular;
  it has no SIRI feed. Caught by the guard.
• כפיר (operator_ref 21) — 4,490 planned, 4,341 "cancelled" → 3.3% operated.
  But it does return rows from /siri_rides/list, so cancelled == planned is
  false and the guard does not fire. It would be scored 0.00 and ranked dead
  last in a national table.
• תנופה line 41 — 847 planned, 846 "cancelled" (0.1%). Same problem, one line
  rather than a whole operator.

3 of the 52 lines sampled were in this state. Whether the root cause is feed coverage
or the SIRI→GTFS matching described in #1775 was not established here — that is
itself worth investigating. Related: #1780 (operators absent from SIRI), #1788 /
#1802 (coverage measurement).

A workable floor: exclude an operator-day whose operated rate is below some threshold
(the sampled genuinely-bad lines bottom out around 68%, while the feed-dead ones are
under 4% — there is a wide gap to put a line in), and cross-check against
network-wide SIRI volume for that day.

Getting started

• Skills: Python or TypeScript, plus a willingness to argue about metric design.
  The statistics involved are averages and rank correlations — nothing exotic.
• The data, hands-on: /rides_execution/list requires all four of line_ref,
  operator_ref, date_from, date_to. Resolve line_refs from /gtfs_routes/list
  for the window you are scoring — one "line" is many line_refs (line 480 was 2
  in Nov 2025 and 8 in Jul 2026), and the set changes over time. No API key needed;
  the checked-in .env already points at production.
• Reference implementation: orion/days_with_no_cancellations.py
  — methods 1 and 2, ~380 lines, with the five live data caveats documented inline.
  Read orion/days_with_no_cancellations.md
  first; it explains why /rides_execution/list was chosen over two alternatives.
• Known cost: one paged request per line_ref. ~80 for a small company, ~1,240
  for אגד. A national ranking is a batch job, not a live query — which is part of
  why Pre-aggregate derived arrival times so these analyses can run interactively #1803 (pre-aggregation) matters.
• Do not average None as 0.0. The reference code returns None (not 0.0) for
  "no planned rides in the window" specifically so this averaging step cannot go
  wrong. Lines with no planned rides must be excluded from the denominator, not
  scored zero.

Related tickets

#1795 (method 1, per line) · #1796 (method 2, per operator) · #1794 (worst days) ·
#1775, #1780 (why actuals go missing) · #1803 (pre-aggregation, the shared cost
bottleneck)

---

Credit & provenance
Found during the hasadna Open Bus hackathon, July 2026 — analysis by the hackathon team.
· Method, evidence and caveats: algorithms/upstream-issues.md
· Original work: https://github.com/hasadna/open-bus-hackathon-26
· Issue drafts and the full defect list: algorithms/upstream-issues.md

Rewritten 2026-08-16 to be readable standalone, and to record the live-API check of
@arielvino's objection. The measurements in "Why this is on hold" were run by an AI
agent against the production Stride API on 2026-08-16 and are reproducible from the
description above — but they are a 52-line sample of small operators, not a census,
and have not been independently reproduced.

[arielvino]

If i understand the idea correctly - it sounds arbitrary and misleading.

A day without cancellation is a binary question.
You don't know if the outage was one single missing ride, or a 50% of all planned rides, for example. So it can flag operators with constant small misses as worse than operators with frequent severe outages, just because the first is smeared upon more calendar dates.

[NoamGaash]

@claude

1. this ticket (and others) doens't have enough context to be understood without reading other materials. please find all of those tickets and make sure it's clear for new contributor what should be done, what's the suggested change, why it's worth their efforts and so on WITHOUT relying on them reading the entire history of everything related to it.
2. Share your opinion - I added "on hold" for now, Aviv's comment sounds reasonable but I'm not sure I fully understand this ticket myself so I'm not comfortable sharing an opinion on it. That's bad.

Thanks!

[claude]

Claude finished @NoamGaash's task in 30m 11s —— View job

---

(I'm an AI agent — Claude. Flagging that up front per this repo's AI-agent disclosure rule.)

• Inventory the tickets that can't be read standalone — exactly 5: Aggregate segment reliability into a per-operator punctuality metric #1799, Aggregate bunching CV into a per-line and per-region regularity score #1800, Rank operators by days-without-cancellations (the unbuilt "method 3") #1801, Per-region SIRI coverage scorecard #1802, Pre-aggregate derived arrival times so these analyses can run interactively #1803
• Read the upstream hackathon materials
• Verify against the live Stride API
• Assess @arielvino's critique
• Rewrite all 5 ticket bodies
• Share an opinion

---

1. The five thin tickets — rewritten

They weren't thin by accident: all five came out of the same scale() template in the hackathon repo, which emitted What / Why / Sketch / Depends on and nothing else. The result referred to "the original code", "the first two methods", "the first-ping issue in this milestone" — without naming a single file or issue number. Nobody could act on that cold.

All five now open with a plain-language explanation of the underlying analysis, link their sibling tickets by number, point at real file paths, list concrete acceptance criteria, and say what skills you need and where to start. #1802 and #1800 also got a "good first contribution that isn't the whole ticket" — both are self-contained fixes that improve an existing chart today.

	now says
#1799	segment reliability rollup — what the ratio means, why single-line views can't answer the operator question, blocked on #1803
#1800	bunching CV rollup — what bunching is and why headway ≠ punctuality, blocked on #1778 with an explanation of why a variance metric is uniquely vulnerable to a bad departure proxy
#1801	see below
#1802	SIRI coverage scorecard — reframed around a concrete example, plus the "this is a feed-quality metric, not a service metric" warning
#1803	pre-aggregated arrival times — leads with the live measurement showing the stop-level actuals are empty

2. My opinion on this ticket: @arielvino is right, and it's measurable

In plain terms, what this ticket proposes: score each bus line by "what fraction of the last 15 days had zero cancellations" — one missed trip and the whole day scores 0 — then average those scores per operator and rank the operators against each other.

@arielvino's objection is that a binary day flag can't tell one missing bus from half the timetable collapsing. I checked it against the production API: 52 lines, 9 operators, 2026-08-01 → 08-15, using the hackathon's own scoring code, with a volume-weighted rate computed from the same rows.

It's worse than a theoretical concern. Two lines, near-identical real service, scored 6.7× apart:

line	rides/day	trips actually operated	binary score
תנופה 11	26	89.7%	0.47
בית שמש אקספרס 12	113	87.8%	0.07

The score tracks frequency, not just reliability — Spearman correlation with rides/day is −0.69. Lines under 20 rides/day average a score of 0.90; lines at 60–119/day average 0.41 — while actually operating 99.2% and 95.6% of their trips respectively.

And it propagates straight into the ranking, which is the entire point of this ticket:

operator	mean-of-binary (what this ticket would publish)	trips actually operated
תנופה	0.53	85.2%
בית שמש אקספרס	0.37	95.1%

תנופה delivers 10 points less service and ranks above. Its average is propped up by two low-frequency lines scoring 1.00 and 0.93. A national table built this way would partly be measuring which operators happen to run rural routes — published as an accountability metric.

Where I'd push back on Aviv slightly: the metric isn't junk. Within one frequency band it discriminates cleanly — at ~230 rides/day, תבל scores 0.80/0.73 (99.8% operated) while סופרבוס scores 0.20/0.27 (96.4%). Cancellations genuinely cluster on bad days rather than scattering randomly. So it's fine as the per-line diagnostic it already is in #1795. The failure is specific to cross-line and cross-operator comparison — which is the only thing this ticket does.

My recommendation: keep it on hold, but reframe rather than close. If the goal is operator accountability, publish a volume-weighted execution rate — pooled 1 − Σcancelled/Σplanned. Same API rows, zero extra cost, comparable across operators, and it's what the hackathon author themselves recommended in their own criticism section. Keep the binary score as a secondary "clean days" column, labelled as not comparable across lines.

3. Something I'd flag as more urgent than the metric design

The ticket lists a feed-health floor as a dependency. I think it's understated:

• כרמלית — 2,040 planned rides, 2,040 "cancelled". It's the Haifa funicular; it has zero rows in /siri_rides/list. The existing cancelled == planned guard catches this one.
• כפיר — 4,490 planned, 3.3% operated. But it does appear in /siri_rides/list, so cancelled == planned is false and the guard doesn't fire. It would score 0.00 and be ranked worst operator in Israel.

3 of my 52 sampled lines were in this state. Whichever metric wins, that has to be solved first, and the existing guard isn't enough. Whether the cause is feed coverage or the ride-matching in #1775, I did not establish.

---

Root cause (of the "on hold" doubt): the binary per-day score is confounded by service frequency, so averaging it across lines ranks operators partly by route mix rather than reliability. · Evidence: live /rides_execution/list + /siri_rides/list queries over 2026-08-01→15, 52 lines / 9 operators, scored with the hackathon's own logic; correlations, band table and operator table above, all reproducible from the method in #1801. · Confidence: high that the frequency confound is real and large; medium on the specific operator ordering — my sample is 52 alphabetically-strided lines from small operators (I skipped אגד and דן on request cost), not a census. · Couldn't verify: whether the near-zero-actual operators are a feed-coverage or a ride-matching failure; whether the effect holds on the large operators; anything about a single 15-day window generalising.

If it'd help, I'm happy to run the same check across אגד/דן, or prototype the volume-weighted alternative so there's something concrete to compare. And @arielvino — your comment is the reason this got checked at all; worth saying that on the record.

No code changes and no PR — this was issue-body edits only.