Skip to content

Fix issues found testing staging uploads - #167

Merged
meomancer merged 10 commits into
mainfrom
fix/staging-test-findings
Oct 15, 2026
Merged

Fix issues found testing staging uploads#167
meomancer merged 10 commits into
mainfrom
fix/staging-test-findings

Conversation

@zamuzakki

@zamuzakki zamuzakki commented Oct 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This is a PR for https://github.com/kartoza/GeoHosting/issues/1108

Fixes found while testing CloudBench's manual + on-demand uploads against staging (GeoNode, GeoServer, PostGIS, S3). Branched off main; backend unit suite + 500 frontend tests pass, tsc/eslint clean.

1. GeoNode ignored a dataset's title and abstract

GeoNode drops dataset_title=/title= on /api/v2/uploads/upload. New apply_dataset_metadata PATCHes /api/v2/datasets/<pk> after import, in a daemon thread so the upload returns immediately.
image

2. Path traversal in the chunked upload (security)

Client filename flowed unsanitised into upload_dir / session.filename, so ../x wrote outside the session dir (authenticated write-anywhere). New _safe_filename (basename only, reject empty/./..) at init and assembly. Present on main today.

3. Missing GeoServer connection returned 500 instead of 404

get_geoserver_client now translates the manager's ValueError into GeoServerError(404), which the views already handle.

4. COG map preview was blank

The preview API never returns bounds for rasters, so the COG was read but never placed. The frontend now reads the extent from the COG itself (geotiff.js), handling EPSG:4326 and the _3857 copy.
Before
image
After
image

5. Text/README preview failed with "Failed to fetch" (CORS)

It fetched a presigned bucket URL (no CORS headers). Now reads through the same-origin /api/s3/proxy/ route (new getS3ProxyURL).
Before
image
After
image

6. Published COG + PMTiles previews were CORS-blocked

The _3857 COG and PMTiles (Map Explorer) were fed presigned URLs too. Now via the proxy. Also: the fetch patch dropped pmtiles' Range header (rebuilt via new Headers()), and getPmtilesStyle now proxies the saved style JSON.
Before
image
After
image
image

7. Point-cloud (COPC) preview was blank — fixed without bucket CORS

A .copc.laz was fetched from the bucket origin in a Web Worker (no CORS, can't set an auth header). Now streamed through the proxy with an absolute URL and the token as ?token=; new QueryTokenAuthentication (signed SSO or DRF token) authenticates it, scoped to S3ProxyView. Verified: Worker range requests return 206, point cloud renders.
Before
image
After
preview-copc

Also in this PR

  • Staging browser tests (tests/staging/, Playwright) covering the four targets + S3 on-demand conversions (COG, PMTiles+GeoParquet, COPC, mosaic); skips cleanly when not logged in, never deletes from S3. Includes a GDAL data generator.
  • User & developer docs (docs/), wired into the mkdocs nav, with UI screenshots.

Out of scope (tracked separately): the .tif (non-_3857) viewer uses tile.openstreetmap.org → 403 (OSM policy); deferred, fix parked on backup/basemap-openfreemap.

@zamuzakki @claude
Fix three issues found testing staging uploads
8603453 
Found while testing CloudBench's manual + on-demand uploads against
staging and trying to break them.

- GeoNode ignored an uploaded dataset's title and abstract (the dataset
  was named after its file or layer). Confirmed by calling GeoNode
  directly: it drops both on /api/v2/uploads/upload, so set them with a
  PATCH once the import finishes. New GeoNodeClient.apply_dataset_metadata
  (+ get_execution/update_dataset), run in a daemon thread from the upload
  view so the request still returns straight away.

- Path traversal in the chunked upload: the client-supplied filename flowed
  into `upload_dir / session.filename`, so `../x` wrote outside the session
  directory (an authenticated write-anywhere). Reduce the name to its
  basename and reject empty/./.. at init and at assembly (_safe_filename).

- A missing GeoServer connection id returned 500: get_client raised
  ValueError but the views only catch GeoServerError. Translate it to
  GeoServerError(404) in get_geoserver_client, as its docstring already says.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki
zamuzakki requested a review from meomancer October 14, 2026 03:59
@zamuzakki
zamuzakki marked this pull request as draft October 14, 2026 05:08
@zamuzakki

zamuzakki commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Found another issues. Will update the PR

@zamuzakki @claude
Fix S3 COG map preview and text preview from the bucket
3975ee9 
Two preview bugs found checking an on-demand conversion's output on
staging.

- A COG/GeoTIFF preview was blank on the map: the preview API only
  returns bounds for GeoParquet, never for rasters, and the frontend
  places the raster only when bounds exist. Read the extent from the COG
  itself (geotiff.js getBoundingBox + geokeys), handling EPSG:4326 and the
  3857 copy CloudBench also writes, and fit the map to it.

- A README/text file failed with "Failed to fetch": the panel fetched a
  presigned URL on the bucket's own origin, which has no CORS headers, so
  the browser blocked it (opening the link directly works). Fetch through
  the same-origin /api/s3/proxy/ route instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki zamuzakki changed the title Fix three issues found testing staging uploads Fix issues found testing staging uploads Oct 14, 2026
zamuzakki and others added 6 commits October 14, 2026 19:39
@zamuzakki @claude
Fix PMTiles map preview: proxy, Range header, saved style
8ad98a2 
The COG preview fix routed COGs through the same-origin proxy, but PMTiles
(and a layer's saved vector style) still hit CORS and a stricter byte-serving
check. Testing the conversions on staging surfaced three more:

- layerLoader: fetch COG and PMTiles through the same-origin proxy, not a
  presigned bucket URL (the bucket's origin sends no CORS headers). COPC
  stays on presigned — its point-cloud reader runs in a Web Worker the
  window.fetch patch can't reach, so the proxy would answer it 401.
- common.ts: build the auth-injecting fetch's headers via Headers() so a
  caller's Range header survives. Spreading a Headers instance dropped it
  (no enumerable own properties), so pmtiles' ranged request lost its Range,
  came back 200, and pmtiles rejected it ("content-length exceeding request").
- mapExplorer.getPmtilesStyle: fetch the saved style JSON through the proxy
  too, so a PMTiles layer renders with its saved colours instead of silently
  falling back to the default.

mapExplorer.test.ts updated for the proxy style fetch (11 pass).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki @claude
Add staging browser tests and a test-data generator
dbd690c 
A pytest suite that drives staging CloudBench through its real UI to exercise
each connector and the S3 on-demand conversions end to end, plus a GDAL
generator for the sample data they upload.

- tests/staging: smoke (01), GeoServer (02), PostGIS (03), GeoNode (04), and
  the S3 on-demand conversions — GeoTIFF->COG (05), a corrupt file failing
  cleanly (06), vector->PMTiles+GeoParquet (07), LAS->COPC (08), tiles->COG
  mosaic (09), and GeoPackage two-job, skipped as deferred on-demand (10).
  Everything is discovered from staging itself; .env.staging holds only
  switches. Each on-demand job starts and deletes one Hetzner server, and
  nothing in S3 is ever deleted.
- manual-test-data/generate.py: synthetic Bali-area vector/raster/point-cloud
  files (and invalid ones) for uploading by hand or in the tests.
- .env.staging.example: the staging test config template.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki @claude
Add user and developer docs: overview, on-demand conversions, more
fa33af8 
New pages, wired into the mkdocs nav.

User Guide
- Overview & Workflow — what CloudBench is, the four connection types, the
  end-to-end flow, and where it runs (standalone vs inside GeoHosting).
- Connections — adding, testing, and editing each connection type.
- On-demand Conversions — the Hetzner per-job conversion feature: the
  source/output matrix, how to run one, the job stages, cost, cancelling, and
  current limits (GeoPackage inspect deferred on-demand, COPC preview needs
  bucket CORS).
- Cloud-Native Formats — COG, PMTiles, GeoParquet, COPC, and when to use each.
- GeoNode Uploads — uploading datasets and how the title is applied after import.
- Troubleshooting — sign-in, connections, uploads/conversions, and preview.

Developer Guide
- Testing — the unit tests plus the tests/staging browser suite (setup, what
  each test covers, the per-job Hetzner cost, and the never-delete-S3 rule).

Also cross-link uploads.md to the on-demand page and correct its stale
GeoPackage section, and add a "start here" pointer on the home page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki @claude
Wire the staging tests into gitignore and pytest config
3cd9662 
- .gitignore: ignore the staging suite's local files — .env.staging, the saved
  auth/browser-state/token, and the .staging work dir (asset cache, screenshots).
- pyproject.toml: register the `staging` pytest marker the suite tags itself with.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki @claude
Add UI screenshots to the user guide
3060b27 
Four screenshots captured from staging, each embedded in the section it
illustrates:

- overview.md — the dashboard (connection tree + server status)
- connections.md — the Add S3 Connection dialog
- on-demand-conversions.md — the S3 upload dialog with the server-type
  picker and per-hour price
- geonode.md — the GeoNode upload dialog (Dataset/Document, title, abstract)

The dashboard's staging DB hostnames are blurred. Map-preview shots (COG /
PMTiles) are deferred until the preview fix on this branch is deployed to
staging, where they can be captured rendering correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki @claude
Move manual-test-data under tests/staging
bed3edd 
The sample-data generator belongs with the staging tests that use it, so
tests/staging is self-contained. Updates the path in conftest (now relative to
the test dir), the testing docs, .env.staging.example, and the generator's own
usage lines. generate.py still writes next to itself, so output lands in the
new location automatically.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki
zamuzakki marked this pull request as ready for review October 15, 2026 04:17
@zamuzakki
zamuzakki marked this pull request as draft October 15, 2026 07:23
zamuzakki and others added 2 commits October 15, 2026 14:35
@zamuzakki @claude
Stream COPC preview through the proxy, not a presigned bucket URL
e921070 
The point-cloud (COPC) preview fetched the .copc.laz straight from the bucket's
own origin, which sends no CORS headers, so the browser blocked it ("NetworkError
when attempting to fetch resource") and the preview stayed blank. The fix routes
it through CloudBench's same-origin proxy like COG/PMTiles — no bucket CORS to
set up.

The COPC reader (maplibre-gl-lidar) runs in a Web Worker, which the app's
window.fetch token-injection patch can't reach and which can't set an
Authorization header. So:

- Backend: `QueryTokenAuthentication` authenticates the token from a `?token=`
  query parameter, accepting either a signed SSO token (GeoHosting iframe) or a
  DRF auth token (standalone login) — the same two the header path takes. Scoped
  to `S3ProxyView` only.
- Frontend: both preview paths (`S3LayerPreview` and `MapExplorer/layerLoader`)
  pass the proxy URL with `?token=`, resolved to an absolute same-origin URL
  (the reader parses it with `new URL(source)`, so a root-relative `/api/...`
  would fail).

Verified against a real COPC: the Worker's byte-range requests return 206 and
the point cloud renders. Drops the "COPC needs bucket CORS" limitation from the
docs. Also fixes a pre-existing branch test (apiSweep.test.ts) that asserted
init.headers as a plain object after common.ts switched to a Headers instance.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki @claude
Add a COPC point-cloud preview screenshot to the docs
5ae4bcc 
Shows the 3D LiDAR viewer rendering a converted .copc.laz, in the
Cloud-Native Formats guide.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@zamuzakki
zamuzakki marked this pull request as ready for review October 15, 2026 08:21
@meomancer

meomancer commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

LGTM, thank you @zamuzakki

@meomancer
meomancer merged commit e5de177 into main Oct 15, 2026
9 checks passed
zamuzakki added a commit that referenced this pull request Oct 22, 2026
@zamuzakki
Merge main into feat/s3-upload-passthrough
b23a63c 
GeoNode's multi-layer GeoPackage upload from main (#170) needs the whole
file on disk, so GeoPackage datasets stay staged; their views now check
the session is the user's and stream each layer from disk. Everything
else is still relayed, and relayed datasets get their title/abstract
set once GeoNode has imported them (#167).
zamuzakki added a commit that referenced this pull request Oct 22, 2026
@zamuzakki
Merge main into feat/upload-passthrough
f6e2e9f 
GeoNode's multi-layer GeoPackage upload from main (#170) needs the whole
file on disk, so GeoPackage datasets stay staged; their views now check
the session is the user's and stream each layer from disk. Everything
else is still relayed, and relayed datasets get their title/abstract
set once GeoNode has imported them (#167).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@meomancer meomancer Awaiting requested review from meomancer

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@zamuzakki @meomancer