Skip to content

perf(export): stream multi-page image exports to a folder on desktop - #874

Open
albertlast wants to merge 1 commit into
mayocream:mainfrom
albertlast:perf/stream-image-export
Open

perf(export): stream multi-page image exports to a folder on desktop#874
albertlast wants to merge 1 commit into
mayocream:mainfrom
albertlast:perf/stream-image-export

Conversation

@albertlast

@albertlast albertlast commented Jul 25, 2026

Copy link
Copy Markdown

Motivation

I hit this on a real project of ~1000 images: exporting the rendered pages runs out of memory and the export never completes. The failure happens before the app even asks where to save, so there is no partial result to fall back on — the whole export is lost.

This PR fixes that for the desktop build, which is where I ran into it.

Known limitation: the browser build very likely still has the same problem. It keeps the existing path — the server buffers every PNG, zips it, and hands over one blob — so a large enough project should still exhaust memory there. The streaming rewrite needs a folder picker and direct per-file writes; on the web that would mean the File System Access API, which is Chromium-only, so it is a different design decision rather than a port of this one. I have not tried to solve that here; the desktop case is the one I actually hit and the one I can test.

What changed

Exporting rendered/inpainted pages used to build every PNG server-side, zip that into a second in-memory copy, ship the whole thing to the client as one blob, and only then open the folder dialog. On large projects that ran out of memory before the user was ever asked where to put the files.

On desktop the flow is now streaming: pick the destination folder first, then export one page per request and write it straight to disk. Peak memory is a single page regardless of how many pages the project has.

Two supporting server-side changes come along, both needed because a folder export now issues one request per page instead of one per export:

  • export_current_project compacts only for the .khr archive, which is the sole format that reads the project directory. Image and PSD exports read the in-memory scene plus blobs that BlobStore::put_bytes already wrote eagerly, so compacting there meant a full-scene encode plus atomic write on every request. Nothing durable is lost — autosave already compacts 500 ms after the last edit, with a 30 s ceiling.
  • png_bytes_for_page reads under the scene lock instead of cloning the whole scene via scene_snapshot(), which made an export O(pages²). The guard is dropped before the decode so the lock is not held across image work.

User-visible behaviour differences

Desktop only, and only for multi-page rendered / inpainted exports:

  • The folder picker opens before any rendering starts, instead of after the entire export was already buffered in memory.
  • A progress card appears in the activity bubble showing File N / M plus the current filename, with a Cancel button. Cancel aborts the in-flight request rather than waiting out the current page.
  • Pages that lack the requested layer are skipped, but keep their position in the numbering — filenames stay page-NNN-<id>.png, gaps included, so a folder export is what the zip contained.
  • If no page has the layer at all, the user gets a message instead of a folder dialog for an empty export.
  • New i18n keys under operations.*, translated for all nine locales.

Unchanged everywhere else. The browser build still takes the existing blob path, as do khr, psd, and single-page exports on every platform. The two Rust changes are shared with the web build but are behaviour-neutral — they remove redundant work, they don't change what any endpoint returns.

How it was verified

  • Manual, on the project that motivated this: exported a ~1000 image project to a folder from the desktop build. It completed instead of running out of memory, the progress card tracked files as they were written, and cancelling mid-run stopped the export.
  • bun run --filter ui test — 163 tests / 23 files pass, including 7 new cases in ui/tests/lib/io/pagesIo.test.ts covering: the picker opening before the first request, no work at all on cancelled picker, index-preserving skip of pages without the layer, an explicit page subset in caller-given order, mid-run cancellation tearing the card down and resetting the flag, the empty-selection message, and continuing past a failed page while reporting the failure count.
  • cargo clippy -p koharu-rpc --all-targets -- -D warnings — clean.
  • cargo fmt -- --check — clean.
  • bun lint:ui — clean; the remaining warnings are pre-existing and in files this PR does not touch.

AI usage disclosure

Per the contributing guide: AI assistance (Claude) was used in producing this patch. It has been reviewed and tested by me before submission.

Exporting rendered/inpainted pages built every PNG server-side, zipped
that into a second in-memory copy, shipped the whole thing as one blob
and only then opened the folder dialog. On large projects that OOM'd
before the user was ever asked where to put the files.

On desktop the UI now asks for the destination folder first, then
exports one page per request and writes it straight to disk, so peak
memory is a single page regardless of project size. A progress card in
the activity bubble reports file N/M and can cancel mid-run, aborting
the in-flight request rather than waiting out the current page.
Filenames keep the server's `page-NNN-<id>.png` numbering, gaps
included, so a folder export matches what the zip contained.

Two supporting server-side fixes, both needed because a folder export
now issues one request per page instead of one per export:

- `export_current_project` only compacts for the `.khr` archive, which
  is the sole format that reads the project directory. Image and PSD
  exports read the in-memory scene plus blobs the blob store already
  wrote eagerly, so compacting there meant a full-scene encode plus
  atomic write on every request. Nothing durable is lost: autosave
  already compacts 500ms after the last edit.
- `png_bytes_for_page` reads under the scene lock instead of cloning
  the entire scene, which made an export O(pages^2). The guard is
  dropped before the decode.

Browser behaviour is unchanged: the web build still takes the existing
blob path, as do khr, psd and single-page exports on every platform.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for your first PR to Koharu.

Please review our contribution guide before review:
https://koharu.rs/contribute/introduction/

In the PR description, include:

  • what changed
  • any user-visible behavior differences
  • how you verified the change

If AI helped produce the patch, a human still needs to review and understand it before submission.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant