Skip to content

provider_design.md

Latest commit

 

History

History
114 lines (87 loc) · 5.03 KB

provider_design.md

File metadata and controls

114 lines (87 loc) · 5.03 KB
Raw

Provider Design

The server exposes one stable API over several backends:

germ / macOS app / plugin / Max / local client
  -> FastAPI sidecar
  -> AudioGenerationProvider
  -> Stable Audio Python / MLX / Stability API / mock
  -> WAV + metadata JSON

AudioGenerationProvider is the only interface used by routes. Provider-specific code stays in server/providers.

Required methods:

def is_available(self) -> bool: ...
def list_models(self) -> list[str]: ...
def load_model(self, model_id: str, device: str = "auto") -> dict: ...
def generate(self, request: GenerateRequest) -> GenerationResult: ...
def audio_to_audio(self, request: AudioToAudioRequest) -> GenerationResult: ...
def inpaint(self, request: InpaintRequest) -> GenerationResult: ...
def continue_audio(self, request: ContinueRequest) -> GenerationResult: ...
def load_lora(self, paths: list[str]) -> dict: ...
def set_lora_strength(self, strength: float, lora_index: int | None = None) -> dict: ...

Mock Provider

The mock provider always reports available and writes stereo 44.1 kHz WAV files. It supports all endpoints so dashboard and germ clients can be tested without model downloads.

Stable Audio Python Provider

The Python provider imports:

from stable_audio_3 import StableAudioModel

It loads models lazily with:

StableAudioModel.from_pretrained(model_id, device=device)

For input audio, torchaudio.load() returns (waveform, sample_rate). The provider converts this to (sample_rate, waveform) before passing it into Stable Audio.

Continuation is implemented as inpainting from the end of the source clip to the target duration.

LoRA loading is implemented through the official package helpers when present. If the installed package changes those helper names, the endpoint returns a clear error rather than failing silently.

The lora list on each generation request is authoritative. The Python model keeps loaded adapter objects for reuse, but germ resets every loaded strength to zero before enabling only the requested adapters at their exact indices. This prevents a strain removed from the UI from leaking invisibly into a later render. An omitted strength uses the upstream default of 1.0; step_range is MLX-only.

Cancellation asymmetry: queued Python-provider jobs can be cancelled before they start, but a running Python render may finish normally because the in-process Stable Audio API does not expose a safe mid-render interrupt hook.

Stable Audio MLX Provider

The MLX provider is a subprocess wrapper over the official optimized/mlx/sa3 CLI. It does not run install scripts from a generation request. Installation must happen through scripts/install_mlx_provider.sh.

Supported command shape:

./sa3 --prompt "footsteps on gravel" --dit sm-sfx --decoder same-s --seconds 4 --out output/audio/steps.wav
./sa3 --prompt "ambient drone" --cfg 3.0 --negative-prompt "drums, vocals" --dit sm-music --decoder same-s --out output/audio/drone.wav
./sa3 --prompt "jazz fusion with electric piano" --dit sm-music --decoder same-s --init-audio input.wav --init-noise-level 0.7 --out output/audio/out.wav
./sa3 --prompt "explosive drum break" --dit sm-music --decoder same-s --init-audio input.wav --inpaint-range "4,7" --out output/audio/inpaint.wav

The wrapper captures stdout, stderr, command, and return code in metadata. It also forwards the request steps value to the MLX CLI and honors GERM_PROVIDER_TIMEOUT_SECONDS so a stuck subprocess returns error metadata instead of hanging indefinitely.

Running MLX jobs observe the job cancellation signal and terminate the sa3 process group with SIGTERM, then SIGKILL if needed.

The official MLX CLI accepts one --inpaint-range per call. The wrapper supports multi-region inpainting by running ranges sequentially: range 1 writes an intermediate file, range 2 uses that file as its source, and the final range writes the requested output. Metadata records multi_range_strategy, intermediate_files, commands, return codes, and per-range seeds.

Stability API Provider

The API provider implements Stability's asynchronous Stable Audio 3 API. It submits text-to-audio, audio-to-audio, and inpaint requests, polls the result endpoint, and implements continuation as an inpainted extension. Multi-range inpainting is sequential because the hosted API accepts one mask range per request; metadata records the generation ids, actual returned seeds, and estimated credits.

Set STABILITY_API_KEY to enable it. GERM_STABILITY_API_URL and GERM_STABILITY_POLL_SECONDS override the endpoint and polling interval. Hosted edits accept WAV or MP3 sources between 6 and 380 seconds and use batch_size=1. The hosted API does not accept germ's local LoRA adapters or negative prompt field, so unsupported controls are rejected or recorded explicitly in metadata instead of being silently treated as active. Sequential multi-range hosted edits require an output duration of at least 6 seconds so the first result remains a valid source for the next request; germ validates this before spending credits.