Native MOSS-TTS-Nano inserts mid-sentence silence / runaway (silence-token attractor)

[jiangzhuo]

Summary

The native (Electron Python sidecar) MOSS-TTS-Nano-100M TTS backend intermittently inserts long silences mid-sentence (typically ~0.5–1.5s before the final word) and occasionally runs away into many seconds of silence. The audible effect: "It's a nice day." plays as "It's a nice … [~1s pause] … day."

This is not a transport/playback bug — it is inherent instability of the MOSS-TTS-Nano-100M autoregressive model (a "silence-token attractor"), modulated by the chosen speaker prompt. Filing to track the second-layer mitigation; we are deferring the implementation for now.

Environment

• Native local inference (Electron Python sidecar), issue feat: Native sherpa-onnx and transformers for Electron client #129 / per-stage pipeline [Feature]: Support hybrid Local/Cloud LLM pipeline & granular node configuration for ASR-Translation-TTS #261
• Backend: moss_onnx (sidecar/sokuji_sidecar/tts_backends.py:MossOnnxTtsBackend), vendored ORT runtime sidecar/sokuji_sidecar/moss_tts/ort_runtime.py
• Model: OpenMOSS-Team/MOSS-TTS-Nano-100M-ONNX + OpenMOSS-Team/MOSS-Audio-Tokenizer-Nano-ONNX
• Runtime: onnxruntime-gpu==1.20.1 (CUDA EP), torch 2.11.0+cu128; reproduced on an RTX 4070. Also reproduces on CPU EP.
• generation_defaults (from the model manifest): max_new_frames=375, do_sample=True, audio_temperature=0.8, audio_top_p=0.95, audio_top_k=25, audio_repetition_penalty=1.2, sample_mode=fixed

Reproduction

1. Native provider, ASR=SenseVoice, MT=Opus-MT (zh→en), TTS=MOSS-TTS-Nano (default preset voice Junhao).
2. Produce a short English utterance, e.g. "It's a nice day." / "It's a nice day tomorrow."
3. Audio plays the first words, pauses ~1s mid-sentence, then finishes.

Direct sidecar repro (bypasses the renderer): drive MossOnnxTtsBackend.generate_stream(text), concatenate chunks, and inspect the RMS envelope. The silence is present in the generated samples themselves.

Evidence

Pipeline is NOT the cause

Browser-side timing (debug:tts-timing) for "It's a nice day.":

• All 12 chunks (2.56s of audio) arrived within +686ms.
• The ring-buffer player played continuously with no mid-stream starvation; STARVING fired only at the very end (buffered=0, enqueued=played=61440).

So delivery (sidecar→browser, localhost WS) and playback (ring buffer + AudioWorklet) are clean. A gapless 2.56s buffer that contains a 1.1s silence ⇒ the silence is in the model output.

Waveform confirms model-generated silence

RMS envelope (80ms/cell), 5%-of-peak silence threshold:

• It's a nice day. → speech 0–0.88s ("It's a nice"), silence 0.80–1.92s (1.12s), "day" 1.92–2.40s.
• It's a nice day tomorrow. → 640ms internal silence at 1.20s.

EOS works; generation is stochastic and variable-length

Frame counts (cap max_new_frames=375): "It's a nice day."=32, "…tomorrow."=24, "Hi."=6 — none hit the cap, so EOS terminates. Re-running the same text yields different lengths/silence (do_sample=True). The model emits silence audio tokens and tends to stay in silence until it samples EOS or hits the cap.

Speaker prompt strongly modulates it (default Junhao is worst for English)

"It's a nice day.", 3 trials each, internal-silence runs ≥240ms:

• Junhao (default): 3/3 bad — 2.56s [(800,1120)], 2.72s [(1040,960)], 4.32s [(720,2880)]
• Xiaoyu/Ava/Bella/Saki/Soyo: 0/3 in this round

Broader sweep — 4 English sentences × 3 trials, count with any internal silence:

voice	bad/total	notes
Ava	3/12	best; max dur 2.4s
Bella	5/12	up to 4.5s
Adam	4/12	runaways 28.6s, 22.5s
Nathan	6/12
Xiaoyu	6/12
Saki	6/12	up to 6.3s

So no preset voice is fully clean; Junhao is simply the worst case for English.

Sampling params don't fix it

• Lowering temperature makes it worse (temp 0.2/0.3 → silences up to 30s, hitting the 375-frame cap).
• repetition_penalty sweep on the two best voices: Ava stays ~2–3/12 (rep 1.2/1.35/1.5); Bella erratic (9/12 at rep 1.2 and 1.5, 3/12 at 1.35). No reliable win.

Root cause

MOSS-TTS-Nano-100M (a 100M autoregressive speech-LLM) exhibits a silence-token attractor: once it samples a silence frame it tends to keep emitting silence frames until it randomly samples the audio-EOS token (id 7) or reaches max_new_frames. This surfaces as long mid-sentence pauses and occasional multi-second runaways. The speaker prompt (built-in voice) shifts how often this happens — a language-mismatched prompt (e.g. the Chinese Junhao voice on English text) triggers it almost every time.

This is model-level instability, not a Sokuji pipeline defect.

Proposed fix

Layer 1 (cheap, done/actionable separately): change the default preset voice from Junhao to a clean English voice (e.g. Ava). Reduces the English failure rate from ~100% to ~25%. Eventually superseded by the per-language / user-selectable voice work (voice-clone UX, Plan B).

Layer 2 (this issue — deferred): add silence governance to the MOSS streaming decode path (MossOnnxTtsBackend._iter_chunks in sidecar/sokuji_sidecar/tts_backends.py), voice-agnostic:

• Compress over-long internal silence: measure per-frame RMS on decoded audio; cap any silent run to a small ceiling (e.g. ~250ms) so a sentence plays gaplessly while still keeping subsequent speech (do not early-stop on the first silence — the model may still produce a trailing word like "day").
• Runaway cutoff: once speech has started, if cumulative silence exceeds a cap (e.g. ~1s), force audio-EOS to stop the AR loop — prevents the multi-second hangs (observed up to ~30s).
• Keep short natural pauses (≤ ~250ms at clause boundaries) so prosody isn't flattened.
• Add tests asserting: (a) a synthetic decoded stream with a long internal silent run is emitted with the silence capped; (b) a runaway silent tail triggers the cutoff.

Alternative / complementary

For users needing rock-solid reliability, prefer the non-autoregressive A-class voices (piper VITS etc.), which have no attractor and are stable; keep MOSS as the voice-cloning option with the Layer-2 mitigation applied.

Notes

• Debug instrumentation used to gather this evidence: debug:tts-timing localStorage flag logs chunk-arrival cadence (LocalNativeClient) and ring-buffer starvation (ModernAudioPlayer).
• Related: feat: Native sherpa-onnx and transformers for Electron client #129 (native local inference), [Feature]: Support hybrid Local/Cloud LLM pipeline & granular node configuration for ASR-Translation-TTS #261 (per-stage pipeline).