QVAC-21928: CosyVoice3 native C++/ggml TTS engine (Qwen2 LM + DiT flow + CausalHiFT)#99
Open
mexxik wants to merge 3 commits into
Open
QVAC-21928: CosyVoice3 native C++/ggml TTS engine (Qwen2 LM + DiT flow + CausalHiFT)#99mexxik wants to merge 3 commits into
mexxik wants to merge 3 commits into
Conversation
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
mexxik
force-pushed
the
feat/QVAC-21928-cosyvoice3-tts-engine-scaffold
branch
from
July 21, 2026 15:11
7421780 to
b4275fe
Compare
…w + CausalHiFT)
Adds tts_cpp::cosyvoice::Engine — a pure C++/ggml, CPU implementation of the
Fun-CosyVoice3-0.5B back-half, under the same public Engine API as the sibling
engines (engines/tts/include/tts-cpp/cosyvoice/engine.h).
Pipeline: text -> Qwen2 BPE tokenizer -> Qwen2.5-0.5B speech LM -> speech tokens
-> DiT conditional-flow-matching (10-step Euler) -> mel -> CausalHiFT vocoder
-> 24 kHz PCM.
- Shared core src/cosyvoice_pipeline.{h,cpp} (in-memory, ggml-only) drives both
the Engine and the parity CLIs (cosyvoice-{llm,flow,hift,cli}).
- KV cache for the LM decode (~25x/token, bit-identical output).
- Self-contained voices: voice.gguf bakes the reference transcript
(voice.prompt_text); switching accent is a one-file swap.
- Instruct mode (EngineOptions::instruct_text, cosyvoice-cli --instruct): Chinese
dialect / emotion / speed / volume control, mirroring inference_instruct2.
- GGUF converters (scripts/convert-cosyvoice3-*-to-gguf.py) — LM --dtype
f32|f16|q8_0|q4_0 (1.9 GB -> 928 MB q8 / 758 MB q4); bake/extract/assemble tools.
- Reference tooling under models_evaluation/CosyVoice/ (parity dumps).
Validation (vs the PyTorch reference, identical inputs):
- LM prefill logits, flow_mel, dit_out all at cosine 1.000000.
- HiFT: after a phase-artifact fix (causal nearest-mode source excitation +
ReflectionPad1d), tracks PyTorch at waveform corr 0.996 on the same mel
(was 0.136); removed an audible metallic ring the log-mel gate missed.
- Instruct path: greedy text_ids + speech-token trajectory bit-identical to
inference_instruct2 (Cantonese 400/400, Sichuan 176/176, same EOS step).
End-to-end Whisper-verified intelligible. Quantized LM (q8/q4) runs unchanged
(ggml mul_mat consumes quantized weights natively).
Scope: CPU-only (iteration 1); zero-shot uses a baked voice (native S3 tokenizer
+ CAM++ for arbitrary reference audio is a follow-up); no GPU path; flow/HiFT f32.
Purely additive — Chatterbox/Supertonic/LavaSR/Parler APIs unchanged. Does not
touch the ggml-speech port; consumes it like the other engines. Staged bring-up
+ the numerical bugs found are documented in engines/tts/PROGRESS_COSYVOICE.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mexxik
force-pushed
the
feat/QVAC-21928-cosyvoice3-tts-engine-scaffold
branch
from
July 21, 2026 15:14
b4275fe to
0eb7b44
Compare
| ggml_tensor * o = ggml_mul_mat(c, vt, sc); | ||
| o = ggml_cont(c, ggml_permute(c, o, 0, 3, 1, 2)); | ||
| o = ggml_reshape_3d(c, o, HD, NH, L); | ||
| o = ggml_reshape_2d(c, o, HD * NH, L); |
…t converted to larger type' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
…t converted to larger type' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
QVAC needs an on-device CosyVoice3 TTS engine.
tts-cppshipped Chatterbox andSupertonic; CosyVoice3 adds strong multilingual (esp. CJK) synthesis and a zero-shot
voice-cloning lineage. There was no native (non-PyTorch) CosyVoice3 path.
How does it solve it?
Adds
tts_cpp::cosyvoice::Engine— a pure C++/ggml, CPU implementation of theFun-CosyVoice3-0.5B back-half, under the same public
EngineAPI as the siblingengines (
include/tts-cpp/cosyvoice/engine.h).Pipeline: text → Qwen2 BPE tokenizer → Qwen2.5-0.5B speech LM → speech tokens →
DiT conditional-flow-matching (10-step Euler) → mel → CausalHiFT vocoder →
24 kHz PCM.
src/cosyvoice_pipeline.{h,cpp}(in-memory, ggml-only) drives boththe
Engineand the parity CLIs (cosyvoice-{llm,flow,hift,cli}).voice.ggufbakes the reference transcript(
voice.prompt_text); the engine reads it, so switching accent is a one-file swap.EngineOptions::instruct_text,cosyvoice-cli --instruct):natural-language control of Chinese dialect / emotion / speed / volume, mirroring
CosyVoice3
inference_instruct2(==zero_shotminus prompt speech tokens; theinstruction sits in the LM prompt before
<|endofprompt|>, the baked voice keepsthe timbre). Supports the 17 Chinese dialects + emotion/speed/volume from
cosyvoice/utils/common.py.scripts/convert-cosyvoice3-*-to-gguf.py) — LM supports--dtype f32|f16|q8_0|q4_0(LM: 1.9 GB → 928 MB q8 / 758 MB q4); plusbake-cosyvoice3-voice.py,extract_voice.py,assemble-cosyvoice3-model.py.models_evaluation/CosyVoice/(parity dumps).Validation
Numerically parity-gated against the PyTorch reference on identical inputs:
flow_mel,dit_outall at cosine 1.000000.excitation + a missing
ReflectionPad1d, commit0e4b8df), our HiFT tracksPyTorch at waveform corr 0.996 on the same mel (was 0.136); the residual is
the random unvoiced excitation (different RNG). Removed an audible metallic ring
that the log-mel gate was blind to.
identical inputs), the Qwen2 BPE
text_idsand the generated speech-tokentrajectory are bit-identical to PyTorch
inference_instruct2— verified onCantonese (45/45 ids, 400/400 steps) and Sichuan (45/45 ids, 176/176 tokens, same
EOS step). Tokenization (incl.
<|endofprompt|>), instruct-prompt construction,LM forward, and EOS handling all match exactly.
End-to-end (LM → flow → HiFT) Whisper-verified intelligible. Quantized LM (q8/q4)
verified intelligible with zero engine changes (ggml
mul_matruns quantizedweights natively).
Scope / not in this PR
CPU-only (iteration 1). Zero-shot uses a baked voice; native S3 tokenizer + CAM++
for arbitrary reference audio is a follow-up. No GPU path yet. Flow/HiFT remain f32.
Breaking changes
None — purely additive (new engine + new CLIs + converter scripts). Existing
Chatterbox/Supertonic/LavaSR APIs and outputs are unchanged.
Follow-ups
embed_tokensget-rows gather → quantize it (LM well under 500 MB); flow quant.Reviewer notes
include/tts-cpp/cosyvoice/engine.h.PROGRESS_COSYVOICE.mddocuments the staged bring-up + the 5 numerical bugs found.ggml-speechport; consumes it like the other engines.