[Stable Diffusion] Enable image upscaling for external calls

[NineBallo]

This adds the ability to set upscaling models, per sd-model. For when you hit the /generate /edit endpoints and want it to go through the upscaler as well.

ESRGAN image upscaling for sd-cpp models

Per-model upscaler configuration in ModelOptionsModal with automatic GPU pass-through to sd-cpp models.

What

• Adds an Upscale Model dropdown to the ModelOptionsModal for sd-cpp recipes ("None" + dynamically discovered upscalers filtered by upscaling label)
• Configured via the upscale_model recipe option — persists per-model, not per-session
• When set, generated/edited/varied images are automatically passed through the selected ESRGAN model before the response is returned
• Upscaler runs on the same GPU backend as the main model (Vulkan/ROCm/CUDA) via --backend <device> flag, avoiding CPU fallback

How

• Frontend: Fetches upscaler models from /models?show_all=true, filters by upscaling label, renders dropdown above merge-args checkbox
• Backend: apply_upscale_if_configured() intercepts image responses in handle_image_generations, handle_image_edits, and handle_image_variations, resolves/downloads the upscaler model,
  and runs sd-cli -M upscale
• GPU pass-through: Backend preference flows from main model → global config → auto-detect, mapped to sd-cli device identifiers (vulkan0, rocm0, cuda0)
• Refactored: Extracted ~200 lines of duplicated upscale logic into shared do_upscale() helper + sd_backend_for_recipe() mapping

Files changed

File	Change
src/cpp/include/lemon/backends/sdcpp/sdcpp.h	Added upscale_model backend descriptor
src/cpp/include/lemon/backends/sdcpp/sdcpp_server.h	Added backend param to upscale_via_cli()
src/cpp/include/lemon/server.h	New apply_upscale_if_configured(), do_upscale(), sd_backend_for_recipe()
src/cpp/server/server.cpp	Hook upscaler into image endpoints; refactored shared pipeline
src/app/src/renderer/ModelOptionsModal.tsx	Upscaler dropdown + dynamic model discovery
src/app/src/renderer/recipes/recipeOptionsConfig.ts	upscaleModel option definition
src/cpp/server/backends/sdcpp/sdcpp_server.cpp	GPU device passthrough in upscale_via_cli()

[NineBallo]

This one has a few possibly polarizing decisions, opening for review but just lmk where you want to go with it. I think it makes a lot of sense but idk.

[jeremyfowers]

[AI-assisted review] Automated pre-review from repo-manager — flags for the human reviewer, not a replacement for one.

Attention level: Elevated — any reviewer can take it, but the to-dos below need resolving before approval (2 documentation gap(s))

Documentation gaps

• Add upscale_model to the sd-cpp parameter table in the /v1/load endpoint section of docs/api/lemonade.md.
  • Gap: The new upscale_model recipe option for sd-cpp is not documented alongside the other sd-cpp parameters.
  • Where: docs/api/lemonade.md
  • Why: Documentation for a new or changed feature belongs in the same PR as the code.
• Add upscale_model to the sd-cpp image_defaults example in docs/guide/configuration/custom-models.md.
  • Gap: The image_defaults example for sd-cpp models does not include the new upscale_model option.
  • Where: docs/guide/configuration/custom-models.md
  • Why: Documentation for a new or changed feature belongs in the same PR as the code.

Scope

minor — The PR extends the existing sd-cpp recipe with a new optional upscale_model parameter and hooks it into three existing image endpoints. It does not introduce a new subsystem or a new user-facing surface — the endpoints already existed and the upscaler is opt-in via a new GUI dropdown within an existing modal.

Suggested reviewers

• danielholanda (GUI) — The PR adds a new dropdown control to ModelOptionsModal, a GUI component under the GUI maintainer area.
• kenvandine (Image AI) — The PR adds backend upscaling logic for sd-cpp image endpoints, which falls under the Image AI maintainer area.

Reviewed at head 5707574 by repo-manager pr-review. Regenerate with repo-manager review-pr 2842.

[jeremyfowers]

@NineBallo please see TODOs in the comment above

@ramkrishna2910 I think you should review

[jeremyfowers]

Also @NineBallo please be aware that we have a feature freeze on gui2, and the gui changes will need to go into GUI3. If you can remove the gui features from this PR and rebase them into gui3 in a future PR that would be great.

[jeremyfowers]

see comment

[NineBallo]

@jeremyfowers Oops totally forgot about the docs! Some of these pr's won't be directly usable in that case but I'll strip ui changes out and open up seperate prs.

GUI PR questions:

1. Should I wait or target gui3-staging?
2. Do you want a pr for each, or one for all of my sd changes? (Figure all but just to confirm)

[jeremyfowers]

@jeremyfowers Oops totally forgot about the docs! Some of these pr's won't be directly usable in that case but I'll strip ui changes out and open up seperate prs.

GUI PR questions:

1. Should I wait or target gui3-staging?

Wait for gui3 to merge to main first

2. Do you want a pr for each, or one for all of my sd changes? (Figure all but just to confirm)

@kpoineal - heads up there's 3 PRs adding some minor UI elements to the image gen UI.

addressed in comments

[bconsolvo]

Thanks for picking this up @NineBallo — the gap in #2836 is real, and I agree the do_upscale() extraction is a nice cleanup regardless of where we land on the rest.

Some context that's relevant here: an earlier iteration of the upscaling work did have exactly this design — an sd-cpp_upscale_model recipe option that applied automatically as post-processing. I removed it in favor of the explicit two-step /images/upscale endpoint. The reason for the split was that upscaling shells out to sd-cli as a separate process, and that startup cost is significant. Making it a distinct request meant the UI could show accurate "Generating..." then "Upscaling..." stages with real timing, and could display the original and upscaled images side by side. Folding it back into /images/generations doesn't remove that cost — it hides it as unexplained latency on an endpoint the caller thinks is just doing diffusion.

Concerns I'd like to talk through before this lands:

1. Response contract. A client that requests 512x512 now gets back 2048x2048 with nothing in the response indicating why. That's surprising for an OpenAI-compatible endpoint, and it can break callers that size their UI or downstream processing off the request parameters.

2. The original is unrecoverable. apply_upscale_if_configured() overwrites data[0].b64_json in place, so there's no way to get the pre-upscale image back. That's the capability the two-step flow was specifically built to preserve.

3. No per-request control. The setting is sticky per model, so a caller can't opt out for a single request. Would an optional request-level parameter (defaulting to the model-level setting) cover your use case? That would keep the "just works for external clients" property while leaving an escape hatch.

4. Error handling looks incorrect. do_upscale() signals failure by returning an empty string, and the callers use res.status == 0 / res.status != 0 to decide whether an error response was already written. cpp-httplib initializes Response::status to -1, not 0. If that's right, two things follow:

   • In apply_upscale_if_configured(), a failed upscale_via_cli() returns res.status != 0 → true, so the endpoint silently returns the un-upscaled image as if nothing went wrong.
   • In handle_image_upscale(), the if (res.status == 0) guard never fires, so the existing "ESRGAN upscale failed" 500 is no longer returned. That's a regression on the current endpoint.

   Worth verifying against the vendored httplib header, but I think both paths need an explicit success/failure signal rather than inferring it from res.status.

5. The --backend fix. Can you share the repro? The current code already selects the per-backend sd-cli build via get_backend_binary_path(), so I'd like to understand what was still falling back to CPU before we add a second mechanism on top.

I want us to make that tradeoff on purpose, and to not lose the progress reporting and side-by-side comparison in the process.

[NineBallo]

@bconsolvo

The goal of this is to be optional, opt-in, and per-model, only for external calls. So you can still have the same flow in the webui chat if you'd like; you could still generate an image and call /images/upscale if you'd like; but you can also now use upscaling for chat apps that don't call /images/upscale (most of em). Given that this project is targeting self-hosters I think this makes a lot of sense, there is no case where the latency is unexplained because they set it up. People with true multi-step image workflows will be using comfyui or something anyway, this directly targets the main usecase for this endpoint. Giving users the choice to get higher quality images in generic chat apps without the exponential cost of generating the base at a higher resolution (especially with consumer hardware).

1. Given this would be opt-in I think it's the right path. I see this as an improvement for chat workflows where you might want to iterate on an image. Chat seems to be the target for lemonade and a lot of them don't have a native /image/upscale pathway. The alternative would be to track the upscaling ratio of each model (currently only x4) then generate at res/4. I don't like this as much, because if you called the endpoint wanting a 512x512 image, you want at least the quality of a 512x512 image. Currently we preserve the quality, though that could be switched to resolution. I'm open to discussion there, and do see that as a viable solution if you feel strongly on it.

2. The idea is to preserve the 2 step workflow, this just adds the option to combine them. You can select no upscaler for that sd model (leave it defaulted), then run the /images/upscale as you normally would if that's what you want. This just makes it possible to have a basic image gen pipeline for external chat applications like openwebui that don't expect any of that.

3. Honestly I just noticed the huge gap in general usability for "normal" chat applications. I don't generate a ton of images but it's occasionally nice to be able to make a quick edit to an image here or there, and this is very helpful there. I like the models being setup like presets though, similar to how we have the llm side is handled with per-model settings. We could add an optional setting: upscale=off to disable but I think it's better to use the proposed solution in 1 instead of that.

4. This was a bit rushed cause I've been real busy IRL. The error handling could totally be wrong I'll take a look when I get a chance lol.

5. I spun this all up in an lxc and messed up my gpu passthrough. Found that without --backend it would silently fall back to the cpu if it didn't see a gpu. Given that I told it to be on the GPU I prefer it erroring to silently continuing and adding like 2 mins to the latency. though I did change a few things between tests so I'll go back and re-test if it's needed. It was a mangled setup and a normal user shouldn't have that problem so it honestly may be.

---

Next Steps:

1. I'll run a QA sweep of the old pipeline/workflow since that is a significant concern and ensure everything still works. 2.Fix the error handling.
2. Review the --backend setting.
3. Just gonna wait for how we want to tackle the endpoint compliance