Skip to content

task/WG-321: improve how vector files are imported - #294

Merged
nathanfranklin merged 14 commits into
mainfrom
task/WG-321-fix-importing-for-geojson-shapefiles
Jul 30, 2026
Merged

task/WG-321: improve how vector files are imported#294
nathanfranklin merged 14 commits into
mainfrom
task/WG-321-fix-importing-for-geojson-shapefiles

Conversation

@nathanfranklin

@nathanfranklin nathanfranklin commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Overview:

This PR along with Hazmapper PR migrates vector file ingest from "N features in PostGIS" to a "1 Feature + PMTiles asset" model.

Uploaded vector files (e.g. GeoJSON, shapefile) are reprojected to EPSG:4326, then the geojson is run through tippecanoe to produce a static .pmtiles archive, and stored as a single FeatureAsset (asset_type="vector") on a Feature. The pmtile is served directly to the client (over HTTP range requests).

Note: The GeoJSON-body endpoint for adding individual features (addGeoJSON) is unchanged.

Related Jira tickets:

Summary of Changes:

  • TippecanoeService (geoapi/services/tippecanoe.py): wraps tippecanoe as a subprocess to build a .pmtiles archive from GeoJSON. Uses -zg to guess the max zoom, floored via --smallest-maximum-zoom-guess so sparse/large-extent data (which otherwise guesses zoom 0) still produces adequately-precise tiles. (Related reading is https://felt.com/blog/tippecanoe-display-scale-precision and https://medium.com/fika-blog/the-dark-art-of-vector-map-tiling-b417a3813df5)
  • VectorService.convert_to_geojson (geoapi/services/vectors.py): reads any supported vector format via geopandas, reprojects to EPSG:4326, strips Z coordinates, extracts the bounding box, and writes GeoJSON to a temp dir. Defines SUPPORTED_VECTOR_EXTENSIONS; removes the now-orphaned process_shapefile.
  • FeaturesService.fromVectorFile? converts to tippecanoe and then copies the .pmtiles into the project asset dir.
  • Improve import of shapefile companion files (geoapi/tasks/external_data.py): the single-file Tapis import now lists the containing directory so only companion shapefile files that actually exist are fetched (avoids noisy 404s) ad non-required-file log downgraded to warning.
  • Routes — the deprecated direct HTTP upload route rejects vector uploads with a 400 and points caller to the Tapis import endpoint.

Testing Steps:

  1. Rebuild the worker image so tippecanoe 2.79.0 is installed (built from source in devops/Dockerfile.worker): docker compose build workers (or your local build target).
  2. Use Task/WG-321: fix importing for geojson shapefiles hazmapper#481
  3. End-to-end import: import a GeoJSON / shapefile / GPX from Tapis into a project
  4. Check that it links back to original file in Designsafe

Notes:

  • GeoPackage (.gpkg) — a multi-layer container (and can hold raster tiles), effectively "the 5 shapefiles users used to send, bundled." Our "1 file = 1 Feature" model would silently ingest only its first layer. Proper support means exploding its vector layers into N Features and surfacing any embedded rasters as internal tile layers. So this could be done later
  • Improved styling

TODO

  • file linking (original + current system/path) recorded on the vector asset
  • add link to the hazmapper PR (client rendering + attribute panel)
  • locally test scraping

- Floor tippecanoe's guessed max zoom so sparse/global vector data still
  produces renderable tiles (and simplify geojson_to_pmtiles to its used args)
- Include vector-asset features in the no_asset_vector feature query
- List the containing directory when importing so only existing companion
  shapefile files are fetched (avoids noisy 404s)
- Serve .pmtiles assets with byte-range support and CORS preflight in nginx
- Add a decode-and-inspect test plus a point+polygon fixture
tippecanoe drops features at low zoom by default, leaving a scattered point
layer nearly empty at the world view. Add --drop-rate=1 so every feature is
kept at every zoom (--drop-densest-as-needed still caps oversized tiles), plus
a worker test asserting >900 of the 1000 points survive at each zoom.
@nathanfranklin nathanfranklin changed the title Task/wg 321 fix importing for geojson shapefiles task/WG-321: improve how vector files are imported Jul 18, 2026

@sophia-massie sophia-massie left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nathan and I huddled - found with geoJSON with no metadata, the message "Click a feature on the map to see its attributes." could be confusing because its attributes will not appear after clicking.

Also noticed that the mouse isn't detecting that the pmtiles are clickable even though they are and work correctly i.e. change metadata depending on the point selected.

@nathanfranklin
nathanfranklin merged commit 0d1e23a into main Jul 30, 2026
3 checks passed
@nathanfranklin
nathanfranklin deleted the task/WG-321-fix-importing-for-geojson-shapefiles branch July 30, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants