Skip to content

sync: Update sync scripts to use write API #66

@victorquinn

Description

@victorquinn

Summary

Update all sync scripts to write to the database via the API instead of committing JSON files to the repo. Add streaming GeoJSON export for tile pipeline.

Scripts to Update

  • scripts/sync-ev-charging.ts → use bulk API
  • scripts/sync-power-plants.ts → use bulk API
  • scripts/sync-power-plants-monthly.ts → use bulk API
  • scripts/sync-transmission-lines.ts → use bulk API
  • scripts/sync-pricing-nodes.ts → use bulk API
  • scripts/sync-ba.ts → use bulk API

New Scripts

  • scripts/export-geojson.ts — Streaming batched PostGIS → GeoJSON export
  • Trigger cache revalidation after sync via revalidateTag()
  • Trigger async tile rebuild via GitHub Actions dispatch

Implementation

  • Each sync uses idempotency key: {source}-sync-{date}
  • Sync checks feature flag — if 'json', writes to files; if 'database', writes to API
  • Streaming GeoJSON export in 1000-record batches (prevents OOM for 85K EV stations)

Acceptance Criteria

  • All sync scripts updated with dual-mode support
  • Idempotency keys prevent duplicate writes on retry
  • Cache revalidation triggered after successful sync
  • Streaming GeoJSON export works for all entity types
  • npm run build passes

Spec ref: Section 7
Blocked by: #21 (write API)
Blocks: Nothing (but enables full database operation)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions