Skip to content

Commit 287bed6

Browse files
committed
gbfs/api: roll back /1m + partials config — defer to backfill
Discovered the architectural limitation during prod rollout: pyrmts treats each (tier, cadence) watermark as "data is valid from epoch through watermark.end", with no per-cadence start date. Our /1m cadence partials roll forward in time only (cascade started today), but pyrmts trusts them for all-of-time before the recorded watermark — planner's segment loop walks /1m as a fall-through for ANY output tier, emits partial keys for pre-cascade-start periods, and 404s on the missing parquets. `earliestWatermarks` doesn't help: it propagates "coarser tiers can't start before finer sources" up the ladder, which is correct for the intended dim-coverage use case but exactly wrong for ours (coarser tiers have MORE history via `e`'s pyramid-cascade backfill). `binBudget` cap doesn't help either: the segment-walk loop traverses /1m regardless of output tier choice. Rollback to pre-#130 query routing: - /1m removed from TIERS (in-file comment explains why + recovery path) - `partials` + `partialKey` removed from pyramid config - `binBudget` reverts to user-controlled (no auto-cap) - `AVAIL_1M_EARLIEST` constant deleted What stays running: - Cascade prod cron writes /1m partial sub-shards at `avail-v3/1m/p{cadence}/...` (data accumulates forward) - D1ShardIndex records watermarks per write - station-luc.json in R2 Recovery path (any of these unblocks #130): - pyrmts adds per-(tier, cadence) earliest-watermark support - /1m partials get backfilled to cover from cascade-start back to /2m's canonical end (~6/18), bridging the historical gap - A separate per-cadence "valid since" date stored in D1, threaded into the planner Prod query parity restored: - Fresh /api/avail-v3?from=2026-06-27 → outputTier=2m, 0 records (same as pre-#130: /2m canonical eff is ~6/18, doesn't cover today) - Historical 6/15 → outputTier=2m → fetches /2m parquet (the existing 1102 timeout on big bbox queries is a pre-existing perf issue, not introduced by #130)
1 parent 379be92 commit 287bed6

1 file changed

Lines changed: 21 additions & 41 deletions

File tree

gbfs/api/src/avail_geo.ts

Lines changed: 21 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -65,14 +65,20 @@ const DEFAULT_REDUCER: Reducer = 'mean';
6565
* data is only ~2 months old — those tiers would have 0-2 bins each.
6666
* Add them back here when the YAML grows them.
6767
*
68-
* Base tier `1m` (bin=1min, shard=1d) is served by the avail-v3
69-
* cascading CFW (task #130): /5m cron writes partial sub-shards at
70-
* `avail-v3/1m/p{cadence}/<period>.parquet`; on midnight UTC boundary
71-
* it promotes to canonical `avail-v3/1m/<date>.parquet`. The planner's
72-
* tier+cadence walk picks the freshest reachable cell (partial
73-
* fall-through to /1m@p5min closes the freshness gap to ≤5 minutes). */
68+
* Base tier `/1m` is REMOVED from `TIERS` until backfill catches up.
69+
* Why: pyrmts treats a (tier, cadence) watermark as "data is valid
70+
* for ALL periods up to watermark.end" — assumes back-to-epoch
71+
* coverage. /1m partials roll forward in time only (cascade CFW
72+
* started writing today); periods before cascade-start don't exist
73+
* but the watermark still claims them. Planner's segment loop walks
74+
* /1m as a fall-through for any output tier and emits partial keys
75+
* for non-existent periods → 404. Until pyrmts supports a partial-
76+
* earliest (per-(tier, cadence) start date), OR partials backfill
77+
* to cover from epoch, /1m can't safely be in TIERS. The cascade
78+
* keeps running (writes accumulate); restore /1m here once partials
79+
* cover enough history that an `AVAIL_1M_EARLIEST` gate is reliable.
80+
* See `specs/avail-v3-cascade-cfw.md` and ctbk task #130 followups. */
7481
const TIERS: Tier[] = [
75-
{ name: '1m', bin: '1min', shard: '1d' },
7682
{ name: '2m', bin: '2min', shard: '2d' },
7783
{ name: '3m', bin: '3min', shard: '3d' },
7884
{ name: '5m', bin: '5min', shard: '5d' },
@@ -96,19 +102,12 @@ function makeBaseProps(bucket: R2Bucket): Omit<GeoPyramid, 'dims'> {
96102
return {
97103
storage: parquetBackend(r2Storage(bucket)),
98104
keyTemplate: KEY_TEMPLATE,
99-
// Per `configs/pyramids/avail.yaml#storage.partialKey` — the cascading
100-
// CFW (#130) writes partial sub-shards at this path; the planner
101-
// reads them via watermark fall-through. `{shard}` substitutes to the
102-
// cadence label (e.g. `5min`, `1h`).
103-
partialKey: 'avail-v3/{tier}/p{shard}/{period}.parquet',
104-
// Sub-shard cadence ladder per `configs/pyramids/avail.yaml#partials`.
105-
// Each cadence applies to every tier where `cadence < tier.shard` and
106-
// `cadence % tier.bin == 0`.
107-
// 7d intentionally omitted: 7d is not a multiple of 3d, breaking
108-
// pyrmts's divisibility-chain requirement on the cadence ladder.
109-
// The /7d tier still gets canonical-only coverage (its `shard: all`
110-
// is the only-shard anyway; no partial would shrink it usefully).
111-
partials: ['5min', '10min', '30min', '1h', '3h', '12h', '1d', '3d'],
105+
// `partials` + `partialKey` intentionally omitted here for now —
106+
// only /1m had partials in scope (#130), and /1m is removed from
107+
// TIERS above pending pyrmts support for per-(tier, cadence) start
108+
// dates. Re-enable both when /1m is restored. The cascading CFW
109+
// continues to write partials at `avail-v3/1m/p{cadence}/...` paths
110+
// and record D1 watermarks; the api just doesn't read them yet.
112111
axis: 'time',
113112
binCol: 'dt',
114113
metrics: METRICS.map((name) => ({ name, monoid: 'histogram' as const })),
@@ -249,13 +248,6 @@ function parseInstant(s: string | null): Date | null {
249248
const PYRAMID_NAME = 'avail';
250249
const WATERMARK_CACHE_TTL_MS = 60_000;
251250

252-
// /1m partials are populated forward in time by the cascading CFW (task
253-
// #130); no historical backfill exists. This date gates the planner from
254-
// emitting /1m partial keys for periods before the cascade's first write.
255-
// Update when partial backfill catches up further; ideally replace with a
256-
// D1 `MIN(period_start)` lookup (see TODO at the use site).
257-
const AVAIL_1M_EARLIEST = new Date('2026-06-27T00:00:00Z');
258-
259251
let _shardIndex: ShardIndex | null = null;
260252
function getShardIndex(db: D1Database): ShardIndex {
261253
if (_shardIndex === null) {
@@ -322,20 +314,8 @@ async function serveGeoReduced(
322314
if (from === null || to === null) {
323315
return errorResponse(400, 'from and to query params required (ISO-8601)', cors);
324316
}
325-
const userBinBudget = parsePositiveInt(url.searchParams.get('bin_budget'), 1024);
326-
if (userBinBudget === null) return errorResponse(400, 'invalid bin_budget', cors);
327-
// /1m exclusion for historical queries: pyrmts's `earliestWatermarks`
328-
// can gate /1m from emitting segments before partial coverage starts,
329-
// but `pickTier` runs first and doesn't consider earliest. If /1m
330-
// becomes the output and then emits 0 segments, the planner returns no
331-
// plan (no fall-through to coarser tiers in that direction). Workaround:
332-
// when the query range extends before the /1m partial rollout, cap
333-
// binBudget below /1m's bin count so pickTier picks /2m+ instead.
334-
const rangeMinutes = Math.ceil((to.getTime() - from.getTime()) / 60_000);
335-
const queryExtendsBeforePartials = from < AVAIL_1M_EARLIEST;
336-
const binBudget = queryExtendsBeforePartials
337-
? Math.min(userBinBudget, rangeMinutes - 1)
338-
: userBinBudget;
317+
const binBudget = parsePositiveInt(url.searchParams.get('bin_budget'), 1024);
318+
if (binBudget === null) return errorResponse(400, 'invalid bin_budget', cors);
339319
const cellBudget = parsePositiveInt(url.searchParams.get('cell_budget'), 1024);
340320
if (cellBudget === null) return errorResponse(400, 'invalid cell_budget', cors);
341321

0 commit comments

Comments
 (0)