Drop duplicate SIRI pings before plotting or averaging

Prompted by a good question — whether the vehicle id and timestamp had
actually been checked. They had not. Checking found one of each:

- Vehicle identity is fine. Across 6,326 rides in a sampled window, zero
  span more than one siri_ride__vehicle_ref, so grouping by siri_ride__id
  never blends two buses, and timestamps are monotonic within a ride.
- Timestamps are not. 1,936 of 20,000 rows (9.7%) are exact duplicates of
  (ride, recorded_at_time, lat, lon) — the same physical observation
  reported in overlapping SIRI snapshots.

Those duplicates were not harmless. On schedule-adherence-map each stop's
position is a 1/distance-weighted average of the pings that matched it, so
a duplicated ping carried its weight twice and pulled the measured route
toward whatever happened to be reported twice; matched pings for the
sample line drop 81 -> 73 once deduplicated. On gps-trace-map they
inflated the ping count and added zero-length map segments (13 -> 12).

Deduplicated at both fetch sites, with the dropped count surfaced in the
card's notes rather than silently discarded. The adherence cache key is
bumped so the fix actually takes effect instead of serving pre-dedup data.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: NoamGaash

analyses/gps_trace_map.py

@@ -81,6 +81,15 @@ def run(req: AnalysisRequest):
                    "07:00-11:00 Israel time — try a different line/operator."],
         )
 
+    # SIRI snapshots overlap, so the same physical observation is reported more
+    # than once — measured at ~10% of rows in a sample window. Left in, those
+    # duplicates inflate the ping count and add zero-length map segments.
+    # (siri_ride__id never spans two vehicle_refs — checked across 6326 rides —
+    # so grouping by ride is safe without also keying on the plate.)
+    before = len(pings)
+    pings = pings.drop_duplicates(subset=["siri_ride__id", "recorded_at_time", "lat", "lon"])
+    duplicates = before - len(pings)
+
     # One ride: whichever siri_ride__id has the most pings in the window — same
     # pick the source notebook made ("the ride with the richest trail").
     ride_id = pings.groupby("siri_ride__id").size().idxmax()
@@ -129,6 +138,8 @@ def run(req: AnalysisRequest):
             f"Picked automatically: the ride with the most GPS pings among all "
             f"{pings['siri_ride__id'].nunique()} rides seen for this line/operator "
             "in the sampled window.",
+            f"{duplicates:,} duplicate ping(s) dropped before plotting — the same "
+            "vehicle, instant and position reported in overlapping SIRI snapshots.",
             _CREDIT,
         ],
     )

analyses/schedule_adherence_average.py

@@ -95,7 +95,7 @@ def _load(line_ref: str, days_back: int):
     matches the reference stop sequence and that have GPS. Disk-cached: this is
     2 API calls per scanned day."""
 
-    key = ("v1", line_ref, days_back)
+    key = ("v2-dedup", line_ref, days_back)
 
     def compute():
         ref_date = datetime.date.today() - datetime.timedelta(days=LAG_DAYS)
@@ -177,6 +177,13 @@ def compute():
 
             actual = pd.DataFrame(actual_rows)
             actual["recorded_at_time"] = pd.to_datetime(actual["recorded_at_time"], utc=True)
+            # Overlapping SIRI snapshots report the same physical observation more
+            # than once (~10% of rows in a sampled window). Duplicates are not
+            # harmless here: the map's average position is distance-weighted, and a
+            # duplicated ping carries its weight twice, pulling the measured route
+            # toward whatever happened to be reported twice.
+            actual = actual.drop_duplicates(
+                subset=["siri_ride__id", "recorded_at_time", "lat", "lon"])
             actual = (_largest_group(actual, "siri_ride__id")
                       .sort_values("recorded_at_time").reset_index(drop=True))
             matched.append({"date": day.isoformat(), "start": w_from, "actual": actual})