[zamuzakki]

This is PR for #155

Problem

Uploads to GeoServer and GeoNode were stored on CloudBench first: the chunks, then the assembled file, then the whole file read into memory to send it. CloudBench has only a small volume (a 10Gi PVC on GeoHosting), so one big upload could fill it. CloudBench connects to any GeoServer/GeoNode, so it can't hand the file over through shared storage either.

Change

Uploads are relayed as they arrive, never stored. The browser still sends 5 MB chunks. CloudBench passes them on in one streaming request to the target, with a Content-Length, and nothing touches its disk.

• Across workers. Chunks can reach any of the 4 gunicorn workers (now gthread). The worker holding the request to the target listens on a Unix socket. The others pass their chunks to it through that socket. No Redis or other new service is needed.
• New API:
  • POST /api/upload/relay starts an upload;
  • PUT /api/upload/relay/<id>/chunks/<n> sends a chunk;
  • GET / DELETE /api/upload/relay/<id> returns the status, or cancels.
• Short drops resume. The browser retries a chunk after 1, 2, 4, 8 and 16 s. A drop longer than about 60 s fails cleanly, and the dialog says to start over: GeoServer and GeoNode can't continue a partial upload. Pause is removed.
• Checked before any data is sent: the connection, the workspace, the file type and store name, and the GeoNode size limit (/api/v2/upload-size-limits/).
  • New: a GeoServer store name already used by a store of another type is refused. Before this, GeoServer took a GeoPackage into a shapefile store, replacing its files but not its type, so its layer broke while the upload said "completed". A file of the store's own type still replaces its data.
• GeoNode zips. CloudBench asks each GeoNode how it takes a zipped dataset, using a 1-byte probe that creates nothing. 4.x needs zip_file with a stand-in base_file; newer GeoNodes take the zip as base_file.
• Cancel. Closing an upload dialog (its Cancel or ×) now cancels the upload. Before, it only hid the dialog while the upload carried on.
• Removed:
  • the old store-then-send endpoints and their Celery task;
  • useChunkedUpload.
• Settings:
  • GUNICORN_THREADS (default 8);
  • GUNICORN_MAX_REQUESTS (default 100000);
  • UPLOAD_RELAY_IDLE_TIMEOUT (60 s);
  • UPLOAD_RELAY_RESPONSE_TIMEOUT (1800 s).

Design, rejected options and known limits: docs/dev-guide/upload-passthrough.md.

Testing

• Unit and API tests for:
  • the relay (order, retries, idle timeout, cancel, backpressure);
  • the GeoServer/GeoNode senders;
  • the up-front checks;
  • the dialogs cancelling on close.
• Full suite: 1588 Python tests and 489 vitest tests pass. ruff, black and tsc are clean.
• deployment/upload-test/ is a production-like stack: production settings, 4 gthread workers, and an nginx set up like GeoHosting's ingress. Through it, against a GeoHosting staging GeoNode 4.4 and its GeoServer:
  • GeoServer: GeoTIFF (2 chunks), zipped shapefile (re-uploaded too) and GeoPackage are created and render;
  • a 20 s gap resumes; a 75 s gap fails cleanly with no store left; cancel leaves no store;
  • a bad store name, a missing workspace and a store of another type are refused up front;
  • GeoNode: a zipped shapefile (detected as zip_file), a GeoTIFF and a document are created;
  • CloudBench's data folder stays the same size, byte for byte, throughout.
• Manual test guide: TEST_GUIDE_upload_passthrough.md.

Known limits

• Newer GeoNodes (5.x/master): zipped datasets sent as base_file are untested.
• GeoNode dataset titles (already on main): GeoNode 4.4 ignores a title sent with the upload, so a dataset is titled after its file. Documents keep their title.