perf(export): stream multi-page image exports to a folder on desktop - #874
Open
albertlast wants to merge 1 commit into
Open
perf(export): stream multi-page image exports to a folder on desktop#874albertlast wants to merge 1 commit into
albertlast wants to merge 1 commit into
Conversation
Exporting rendered/inpainted pages built every PNG server-side, zipped that into a second in-memory copy, shipped the whole thing as one blob and only then opened the folder dialog. On large projects that OOM'd before the user was ever asked where to put the files. On desktop the UI now asks for the destination folder first, then exports one page per request and writes it straight to disk, so peak memory is a single page regardless of project size. A progress card in the activity bubble reports file N/M and can cancel mid-run, aborting the in-flight request rather than waiting out the current page. Filenames keep the server's `page-NNN-<id>.png` numbering, gaps included, so a folder export matches what the zip contained. Two supporting server-side fixes, both needed because a folder export now issues one request per page instead of one per export: - `export_current_project` only compacts for the `.khr` archive, which is the sole format that reads the project directory. Image and PSD exports read the in-memory scene plus blobs the blob store already wrote eagerly, so compacting there meant a full-scene encode plus atomic write on every request. Nothing durable is lost: autosave already compacts 500ms after the last edit. - `png_bytes_for_page` reads under the scene lock instead of cloning the entire scene, which made an export O(pages^2). The guard is dropped before the decode. Browser behaviour is unchanged: the web build still takes the existing blob path, as do khr, psd and single-page exports on every platform. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
albertlast
requested review from
Map1en,
fffonion,
karrot0,
liksunrice and
mayocream
as code owners
July 25, 2026 17:48
Contributor
|
Thanks for your first PR to Koharu. Please review our contribution guide before review: In the PR description, include:
If AI helped produce the patch, a human still needs to review and understand it before submission. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
I hit this on a real project of ~1000 images: exporting the rendered pages runs out of memory and the export never completes. The failure happens before the app even asks where to save, so there is no partial result to fall back on — the whole export is lost.
This PR fixes that for the desktop build, which is where I ran into it.
Known limitation: the browser build very likely still has the same problem. It keeps the existing path — the server buffers every PNG, zips it, and hands over one blob — so a large enough project should still exhaust memory there. The streaming rewrite needs a folder picker and direct per-file writes; on the web that would mean the File System Access API, which is Chromium-only, so it is a different design decision rather than a port of this one. I have not tried to solve that here; the desktop case is the one I actually hit and the one I can test.
What changed
Exporting rendered/inpainted pages used to build every PNG server-side, zip that into a second in-memory copy, ship the whole thing to the client as one blob, and only then open the folder dialog. On large projects that ran out of memory before the user was ever asked where to put the files.
On desktop the flow is now streaming: pick the destination folder first, then export one page per request and write it straight to disk. Peak memory is a single page regardless of how many pages the project has.
Two supporting server-side changes come along, both needed because a folder export now issues one request per page instead of one per export:
export_current_projectcompacts only for the.khrarchive, which is the sole format that reads the project directory. Image and PSD exports read the in-memory scene plus blobs thatBlobStore::put_bytesalready wrote eagerly, so compacting there meant a full-scene encode plus atomic write on every request. Nothing durable is lost — autosave already compacts 500 ms after the last edit, with a 30 s ceiling.png_bytes_for_pagereads under the scene lock instead of cloning the whole scene viascene_snapshot(), which made an export O(pages²). The guard is dropped before the decode so the lock is not held across image work.User-visible behaviour differences
Desktop only, and only for multi-page
rendered/inpaintedexports:File N / Mplus the current filename, with a Cancel button. Cancel aborts the in-flight request rather than waiting out the current page.page-NNN-<id>.png, gaps included, so a folder export is what the zip contained.operations.*, translated for all nine locales.Unchanged everywhere else. The browser build still takes the existing blob path, as do
khr,psd, and single-page exports on every platform. The two Rust changes are shared with the web build but are behaviour-neutral — they remove redundant work, they don't change what any endpoint returns.How it was verified
bun run --filter ui test— 163 tests / 23 files pass, including 7 new cases inui/tests/lib/io/pagesIo.test.tscovering: the picker opening before the first request, no work at all on cancelled picker, index-preserving skip of pages without the layer, an explicit page subset in caller-given order, mid-run cancellation tearing the card down and resetting the flag, the empty-selection message, and continuing past a failed page while reporting the failure count.cargo clippy -p koharu-rpc --all-targets -- -D warnings— clean.cargo fmt -- --check— clean.bun lint:ui— clean; the remaining warnings are pre-existing and in files this PR does not touch.AI usage disclosure
Per the contributing guide: AI assistance (Claude) was used in producing this patch. It has been reviewed and tested by me before submission.