Skip to content

Commit 59d09cb

Browse files
NoamGaashclaude
andcommitted
Fix zig-zagging average on the day-to-day stringline
The "Average actual" line ran backwards along the route in 14 of its 35 points — visually a zig-zag, and physically impossible: a bus cannot reach a later stop earlier. Cause: matched days cover very different subsets of the route (13, 5, 21 and 24 of 41 stops in the sample case) because GPS drops out or trails start late. A naive per-stop mean across days therefore took stop 12 from one set of days and stop 13 from another, describing an "average trip" no bus ever ran. Individual days were nearly monotonic already (0-1 backward steps each), which is what pointed at the averaging rather than the per-ping matching. Two changes: - A stop only enters the average if at least half the matched days (min 2) measured it, so each averaged point rests on a comparable base. - Any residual backward step is clamped forward, since that is nearest-stop mis-assignment rather than a reversing bus. Both are disclosed in the chart's notes — how many thin stops were left out and how many points were clamped — rather than silently smoothed. The excluded stops still appear as faint per-day lines, so nothing is hidden. Verified on real data: backward steps 14 -> 0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 1962322 commit 59d09cb

1 file changed

Lines changed: 36 additions & 2 deletions

File tree

analyses/schedule_adherence_average.py

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -267,10 +267,36 @@ def run_average(req: AnalysisRequest):
267267
name="Planned (GTFS)", emphasis=True, dashed=True,
268268
points=[Point(x=float(v), y=float(i)) for i, v in enumerate(planned_elapsed)],
269269
))
270-
avg = pd.concat([d["elapsed"] for d in per_day], axis=1).mean(axis=1, skipna=True)
270+
# Averaging per stop across days is only meaningful where the same stops were
271+
# measured on comparable days. Days cover very different subsets of the route
272+
# (GPS drops out, trails start late), so a naive per-stop mean can take stop 12
273+
# from a slow day and stop 13 from a fast one — producing an "average" trip
274+
# that runs backwards along the route, which no bus ever did. Requiring a
275+
# minimum number of days per stop keeps each averaged point on a comparable base.
276+
by_day = pd.concat([d["elapsed"] for d in per_day], axis=1)
277+
days_per_stop = by_day.notna().sum(axis=1)
278+
min_days = max(2, (len(per_day) + 1) // 2)
279+
avg = by_day.mean(axis=1, skipna=True).where(days_per_stop >= min_days)
280+
dropped = int(((days_per_stop > 0) & (days_per_stop < min_days)).sum())
281+
282+
# Whatever survives should still climb monotonically — a bus cannot reach a
283+
# later stop earlier. Residual dips are nearest-stop mis-assignment, not the
284+
# bus reversing, so clamp them forward and say how many there were rather
285+
# than drawing a physically impossible line.
286+
avg_points = [(int(i), float(v)) for i, v in avg.items() if pd.notna(v)]
287+
clamped = 0
288+
running = float("-inf")
289+
monotonic: list[tuple[int, float]] = []
290+
for stop_i, val in avg_points:
291+
if val < running:
292+
clamped += 1
293+
val = running
294+
running = val
295+
monotonic.append((stop_i, val))
296+
271297
series.append(Series(
272298
name=f"Average actual (n={len(per_day)} days)", emphasis=True, color="var(--s2)",
273-
points=[Point(x=float(v), y=float(i)) for i, v in avg.items() if pd.notna(v)],
299+
points=[Point(x=v, y=float(i)) for i, v in monotonic],
274300
))
275301

276302
sk = data["skipped"]
@@ -287,6 +313,14 @@ def run_average(req: AnalysisRequest):
287313
f"Days skipped while scanning: {sk['no_plan']} with no timetable, "
288314
f"{sk['route_mismatch']} running a different stop pattern (a different "
289315
f"journey, not missing data), {sk['no_actual']} with no GPS.",
316+
f"The average only covers stops measured on at least {min_days} of the "
317+
f"{len(per_day)} matched days"
318+
+ (f" — {dropped} thinly-covered stop(s) left out of it, though their "
319+
"individual days still show as faint lines." if dropped else "."),
320+
*([f"{clamped} point(s) on the average dipped backwards along the route "
321+
"— nearest-stop mis-assignment, not a reversing bus — and were "
322+
"clamped forward to keep the line physically possible."]
323+
if clamped else []),
290324
f"Only {len(per_day)} days matched — the source notebook required 20 "
291325
"before averaging, so treat this as indicative unless you raise the "
292326
"scan window.",

0 commit comments

Comments
 (0)