You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
gbfs/api: roll back /1m + partials config — defer to backfill
Discovered the architectural limitation during prod rollout: pyrmts
treats each (tier, cadence) watermark as "data is valid from epoch
through watermark.end", with no per-cadence start date. Our /1m
cadence partials roll forward in time only (cascade started today),
but pyrmts trusts them for all-of-time before the recorded watermark
— planner's segment loop walks /1m as a fall-through for ANY output
tier, emits partial keys for pre-cascade-start periods, and 404s
on the missing parquets.
`earliestWatermarks` doesn't help: it propagates "coarser tiers can't
start before finer sources" up the ladder, which is correct for the
intended dim-coverage use case but exactly wrong for ours (coarser
tiers have MORE history via `e`'s pyramid-cascade backfill).
`binBudget` cap doesn't help either: the segment-walk loop traverses
/1m regardless of output tier choice.
Rollback to pre-#130 query routing:
- /1m removed from TIERS (in-file comment explains why + recovery path)
- `partials` + `partialKey` removed from pyramid config
- `binBudget` reverts to user-controlled (no auto-cap)
- `AVAIL_1M_EARLIEST` constant deleted
What stays running:
- Cascade prod cron writes /1m partial sub-shards at
`avail-v3/1m/p{cadence}/...` (data accumulates forward)
- D1ShardIndex records watermarks per write
- station-luc.json in R2
Recovery path (any of these unblocks #130):
- pyrmts adds per-(tier, cadence) earliest-watermark support
- /1m partials get backfilled to cover from cascade-start back to
/2m's canonical end (~6/18), bridging the historical gap
- A separate per-cadence "valid since" date stored in D1, threaded
into the planner
Prod query parity restored:
- Fresh /api/avail-v3?from=2026-06-27 → outputTier=2m, 0 records
(same as pre-#130: /2m canonical eff is ~6/18, doesn't cover today)
- Historical 6/15 → outputTier=2m → fetches /2m parquet (the existing
1102 timeout on big bbox queries is a pre-existing perf issue, not
introduced by #130)
0 commit comments