Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/api/openai.md
Original file line number Diff line number Diff line change
Expand Up @@ -915,7 +915,7 @@ A typical workflow is to generate an image first, then upscale it:

Speech Generation API. You provide a text input and receive an audio file. This API uses [Kokoros](https://github.com/lucasjinreal/Kokoros) as the backend.

> **Note:** Supported models are `kokoro-v1` (fixed voices, [Kokoros](https://github.com/lucasjinreal/Kokoros) backend) and the OpenMOSS family — `OpenMOSS-TTS` (voice cloning from a reference WAV) and `MOSS-VoiceGen` (voice design from a text description).
> **Note:** Supported models are `kokoro-v1` (fixed voices, [Kokoros](https://github.com/lucasjinreal/Kokoros) backend) and the OpenMOSS family — `OpenMOSS-TTS` and `MOSS-TTS-Local`, both of which clone a voice from a reference WAV and can design one from a text description. Voice design is not a separate model: the voice generator ships as a component of the speech model, and a design request swaps it in just long enough to render the reference sample before the speech model is reloaded. That request therefore takes noticeably longer than a plain one, and the result is cached per description.
>
> **Limitations:** `kokoro-v1` supports `mp3`, `wav`, `opus`, and `pcm`; OpenMOSS models natively produce `wav` only, and other formats are rejected with `400 Bad Request`. Streaming is supported in `audio` (`pcm`) mode on `kokoro-v1`.

Expand All @@ -929,6 +929,7 @@ Speech Generation API. You provide a text input and receive an audio file. This
| `voice` | No | The voice to use. All OpenAI-defined voices can be used (`alloy`, `ash`, ...), as well as those defined by the kokoro model (`af_sky`, `am_echo`, ...). Default: `shimmer` | <sub>![Status](https://img.shields.io/badge/partial-yellow)</sub> |
| `voice` (OpenMOSS) | No | For OpenMOSS models the field is a free-text voice/style instruction instead of a fixed voice name (e.g. `a calm, deep male narrator voice`). | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
| `reference_wav_b64` | No | Lemonade extension (OpenMOSS voice cloning): base64-encoded WAV sample of the voice to clone. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
| `voice_design_description` | No | Lemonade extension (OpenMOSS voice design): a description of the voice to invent, e.g. `a warm low female voice with a British accent`. Lemonade renders a short sample in that voice and uses it as the reference, so the effect is the same as supplying `reference_wav_b64` yourself. Ignored when `reference_wav_b64` is also present. Design is opt-in through this field only — `voice` never triggers it. | <sub>![Status](https://img.shields.io/badge/available-green)</sub> |
| `response_format` | No | Format of the response. `mp3`, `wav`, `opus`, and `pcm` are supported. Default: `mp3`| <sub>![Status](https://img.shields.io/badge/partial-yellow)</sub> |
| `stream_format` | No | If set, the response will be streamed. Only `audio` is supported, which will output `pcm` audio. Default: not set| <sub>![Status](https://img.shields.io/badge/partial-yellow)</sub> |

Expand Down
7 changes: 4 additions & 3 deletions docs/dev/backends-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,12 +283,13 @@ the generator instead. Prose outside the markers is preserved. -->
| `Bert-Phishing-ONNX` | 1.34 | classification |
| `Phishing-Email-Detection-ONNX` | 0.27 | classification |

#### `openmoss` — OpenMOSS TTS (2 models)
#### `openmoss` — OpenMOSS TTS (3 models)

| Model | Size (GB) | Labels |
|-------|-----------|--------|
| `MOSS-VoiceGen` | 7.3 | tts, voice-design |
| `OpenMOSS-TTS` | 12.5 | tts |
| `MOSS-SoundEffect` | 6.6 | audio-generation |
| `MOSS-TTS-Local` | 15.8 | tts |
| `OpenMOSS-TTS` | 19.8 | tts |

#### `ryzenai-llm` — Ryzen AI LLM (79 models)

Expand Down
13 changes: 7 additions & 6 deletions src/app/docs/audio-and-3d-modalities.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ GUI3 treats speech input, speech output, generated audio, and 3D as distinct cap
| --- | --- | --- | --- |
| Transcription | `whispercpp`, `moonshine` | audio file or microphone -> text | `/api/v1/audio/transcriptions`, `/v1/realtime` |
| TTS | `kokoro`, `openmoss` | text -> spoken audio | `/api/v1/audio/speech` |
| Audio generation | `acestep`, `thinksound` | prompt -> music or sound effect | `/api/v1/audio/generations` |
| Audio generation | `acestep`, `thinksound`, `openmoss` | prompt -> music or sound effect | `/api/v1/audio/generations` |
| 3D | `trellis` | image -> GLB, or text -> reference image -> GLB | `/api/v1/3d/generations` |

## ACE-Step and ThinkSound
Expand All @@ -15,13 +15,14 @@ ACE-Step exposes music duration, steps, seed, optional structured lyrics, and a

## OpenMOSS

OpenMOSS follows the established GUI2 workflows while fitting the unified GUI3 composer:
OpenMOSS speech has two composer modes, both against a single loaded model:

- **Plain:** synthesize text with an optional style instruction.
- **Describe:** use `MOSS-VoiceGen` directly, or design a short reference voice and pass it to `OpenMOSS-TTS` when both models are installed.
- **Clone:** attach one validated WAV sample and synthesize through `OpenMOSS-TTS` with `reference_wav_b64`.
- **Describe voice:** send the description as `voice_design_description`. The backend invents a voice matching it and speaks the text in that voice.
- **Clone WAV sample:** attach one validated WAV sample, sent as `reference_wav_b64`. The optional style note travels as `voice`, so delivery can be directed without changing the timbre.

Voice-design and clone models are discovered from downloaded or loaded OpenMOSS models. Model switching is performed explicitly between the design and synthesis stages so the second request cannot accidentally run against the first backend instance.
Voice design is no longer a second model the GUI switches to. The voice generator ships as a component of the speech model, and the backend now drives the sequence the GUI used to drive by hand: it takes the speech model down, brings the voice generator up to render one reference sample, tears it down, and brings the speech model back. Only ever one model at a time, so any card that can run the speech model can design a voice for it. The sample is cached per description, so repeating a description costs nothing.

`MOSS-SoundEffect` is an audio-generation model rather than a speech one, exposing steps, CFG, sigma shift, a negative prompt, and a seed. Its seed is unsigned with no random sentinel — `0` is a real seed — so the composer draws a random one whenever the seed box is left blank.

## TRELLIS

Expand Down
13 changes: 13 additions & 0 deletions src/app/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { AccountSession, currentSession, subscribeAccountSessionChanges } from '
import { setPresetStorageScope } from './presetStore';
import { customModelToModelInfo, loadCustomModels } from './features/customModels/customModelStore';
import { findModelInfoByName, isCollectionFullyLoaded, isCollectionModel, withVirtualLoadedCollections } from './features/collections/collectionModels';
import { isModelSelectionLocked, onModelSelectionUnlocked } from './features/modelSettings/modelSelectionLock';
import ChatView from './components/ChatView';
import ModelManager from './components/ModelManager';
import ConnectView from './components/ConnectView';
Expand Down Expand Up @@ -314,10 +315,16 @@ const App: React.FC = () => {
};
setLoadedModels(enriched);
setCurrentModel(current => {
if (current && isModelSelectionLocked()) return current;
if (current && enriched.some(m => m.model_name === current && (canSelectInComposer(m) || customSelectable(m.model_name) || infoSelectable(m.model_name)))) return current;
if (current) {
const info = findModelInfoByName(knownInfos, current);
if (info && isCollectionModel(info) && isCollectionFullyLoaded(info, loaded)) return current;
// A downloaded model the user picked stays picked while it is still
// being prepared. Loading a backend can take minutes (install, then
// weights), and eviction is a resource decision, not a user one —
// neither should silently retarget the composer.
if (info && (info as any).downloaded) return current;
}
const virtualOmni = enriched.find(model => {
const info = findModelInfoByName(knownInfos, model.model_name);
Expand All @@ -330,6 +337,12 @@ const App: React.FC = () => {
});
}, [accountSession.storageScope]);

useEffect(() => onModelSelectionUnlocked(() => {
api.refresh()
.then(result => { if (result) applyLoadedModels(result.health.all_models_loaded); })
.catch(() => { });
}), [applyLoadedModels]);

const navigateToRoute = useCallback((nextRoute: AppRoute) => {
if (nextRoute.view === 'dashboard') lastWorkspaceSectionsRef.current.dashboard = nextRoute.section;
if (nextRoute.view === 'connect') lastWorkspaceSectionsRef.current.connect = nextRoute.section;
Expand Down
Loading
Loading