feat: bump OpenMoss engine to 0.2.0, fix cascade generation, add OpenMOSS SFX model - #2838
feat: bump OpenMoss engine to 0.2.0, fix cascade generation, add OpenMOSS SFX model#2838pwilkin wants to merge 6 commits into
Conversation
Folds MOSS-VoiceGen into the speech models as a `voicegen` checkpoint role rather than a standalone entry — it never took normal input, and a user reaches the same result through the cascade. The cascade therefore moves server-side: OpenMossServer spawns the designer on demand, renders a reference sample, tears it down, and caches by description, so steady-state residency is the speech model alone. - Add MOSS-TTS-Local (local-transformer family, 48 kHz stereo) alongside the delay family, each with its own sampling defaults. - Add MOSS-SoundEffect via IAudioGenerationServer on /sfx. - Expose the speech sampling surface and the SFX solver params in the composer, seeded from per-model speech_defaults / audio_defaults. - Bump the openmoss pin to v0.2.0 with checksums. A backend's descriptor default_labels are now a fallback rather than an override, so openmoss can serve both tts and audio-generation from one recipe without duplicating the binary install. Verified no registered model changes deployment type under the new rule. Also bounds generation length from the input text: the delay family reports n_vq == 32 and so gets no bound from the server, which turned a one-line prompt into 325 s of audio. And sizes --n-ctx/--n-batch for reference-conditioned prefill, which overran the 8192/512 defaults. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZCVEcu9UQa7B5TMX9sXoY
The sidecar carries every moss.* tensor and all model metadata, so a model missing it loads but silently degrades to the wrong architecture and produces no audio. It was fetched only as a side effect of select_checkpoint_files, so it was neither completeness-checked nor counted in the model size. Declaring it as `main_extras` makes are_required_checkpoints_complete refuse a model whose sidecar is absent, and brings reported sizes in line with what is actually on disk (SoundEffect 3.79 -> 6.57 GiB). Dedupe files_to_download, since a file can now be reached both through the backend's select_checkpoint_files and as its own role. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZCVEcu9UQa7B5TMX9sXoY
f263fa4 to
6234a88
Compare
fl0rianr
left a comment
fl0rianr
left a comment
There was a problem hiding this comment.
Thanks for the update! The overall direction makes sense, especially moving the VoiceGen cascade behind the OpenMOSS backend and exposing SoundEffect as a regular audio-generation model.
I found a few functional issues that should be looked into before merging:
- The transient VoiceGen process can leave requests hanging for a very long time
design_reference_sample() polls /health up to 3000 times, but never checks whether the spawned process has already exited. If VoiceGen fails during model loading, for example because both models do not fit in VRAM, Lemonade may continue polling for many minutes instead of returning the actual startup error.
Please reuse the normal backend startup behaviour here: check ProcessManager::is_running(), reap an exited process, and include its exit code/output in the error.
- The new cascade requires TTS and VoiceGen to be resident simultaneously
The main TTS process remains loaded while a second fully GPU-offloaded VoiceGen process is started. Previously the GUI loaded VoiceGen and TTS sequentially, so cards that can run each model individually may now fail because both models plus their context and compute buffers must fit at once.
This likely needs either a memory-aware sequential fallback, explicit placement of VoiceGen on another device/CPU, or another mechanism that avoids requiring both large models in GPU memory simultaneously.
- Any non-empty
voicevalue now triggers voice design
apply_voice_design() interprets the standard /v1/audio/speech voice field as a voice-description prompt, generates a reference WAV, removes voice, and forwards the request.
That changes the existing OpenMOSS/OpenAI-compatible semantics. For example, a normal client sending "voice": "default" would unexpectedly launch VoiceGen and attempt to design a voice called “default”.
Voice design should be enabled through an explicit extension such as voice_design_description or voice_mode: "design", while regular voice values continue to be forwarded as instructions.
- The new
audio_defaultsare not used by GUI3
MOSS-SoundEffect declares 100 steps and CFG 4.0, but the GUI still initializes generic audio generation with 50 steps and CFG 4.5 and sends those values explicitly. As a result, the model defaults are never actually used.
It would be good to initialize the controls from audio_defaults, similar to how speech_defaults are handled.
seed: -1does not mean random for MOSS-SoundEffect
The OpenMOSS SoundEffect seed is an unsigned value and explicitly has no random sentinel. GUI3 currently sends -1 by default, which may either fail conversion or become a fixed large unsigned seed.
For OpenMOSS SFX, please omit the seed when the GUI value is -1, or generate an actual random valid seed.
- The clone-mode “Style note” is currently ignored
The GUI exposes an optional style note in clone mode, but the request path sets voice = '' before sending the cloned reference audio. Either forward the style note as voice, or remove the field from the UI.
The main blockers for me are the child-process lifecycle, simultaneous GPU residency, and changing the meaning of the standard voice field.
…oice` Addresses the three blockers from review. The design child was polled for up to 3000 attempts without ever asking whether it was still alive, so a voicegen that died during model loading left the request hanging for ten minutes instead of reporting the failure. Check is_running(), reap the exited child, and put its exit code in the error, matching what wait_for_ready() already does for the main process. Voice design no longer runs beside the resident speech model. It drives the same sequence the GUI used to drive by hand: the speech model comes down, the voice generator renders one reference sample, and the speech model goes back up. Holding both at once needs a card that fits the pair, which is a strictly harder requirement than running the model the user asked for -- and the pair is 7.25 GB of voicegen on top of the speech model. load() and unload() grow the design lock, since an unload landing mid-swap would otherwise race the restart and leave a live process behind a server that believes it has none. `voice` keeps its OpenAI-compatible meaning. Design is opt-in through a new `voice_design_description` field, so a client sending "voice": "default" gets speech rather than an attempt to invent a voice by that name. Asking for design on a model with no voicegen component is now an error rather than silently synthesising without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tyle note Three GUI-side findings from review. Audio generation initialised steps and CFG to the generic 50/4.5 and sent them explicitly, so the audio_defaults a model declares were never used -- MOSS-SoundEffect asks for 100 steps and CFG 4.0 and got neither. Seed the controls from the model's audio_defaults, letting a preset still win, the way speech_defaults already work. The SFX seed is an unsigned value whose 0 is a real seed, not a "random" sentinel, so the GUI's -1 default was neither. Omitting it would have fallen through to that 0 and rendered a byte-identical clip on every generation; draw a random seed instead, so a blank seed box still means random. Clone mode set voice to '' before sending the reference audio, silently discarding the style note the UI collects. Forward it, so delivery can be directed without changing the cloned timbre. Describe mode sends its description as `voice_design_description` now that `voice` no longer triggers design. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 0.2.0 work dropped MOSS-VoiceGen as a standalone model but left the suite pointing at it, so test_010_voice_design failed at pull time. Design needs no model of its own now: run it against the speech model through `voice_design_description`, and add a case pinning the other half of the contract -- that a plain "voice": "default" speaks rather than designing a voice by that name. Both docs still described the old arrangement. The API reference advertised MOSS-VoiceGen as a supported speech model and had no entry for the new `voice_design_description` extension; the GUI doc described a two-model cascade the GUI drove by hand and a "Plain" mode that no longer exists, and omitted openmoss from the audio-generation recipes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
json::value() throws on a type mismatch instead of falling back to the default, so the frame-bound heuristic turned a numeric `input` into an uncaught type_error and a 500. Validate `input` and `prompt` types in the handlers, and read the backend's string fields type-safely — the collection orchestrator reaches audio_speech() without the handler. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@fl0rianr could you check now? The only error is the macOS CI that never works. |
|
Yeah. |
2 participants
This brings a round of needed fixes to OpenMOSS: