QVAC-21921 tts-cpp: native ACE-Step music generation on CPU#98
QVAC-21921 tts-cpp: native ACE-Step music generation on CPU#98freddy311082 wants to merge 3 commits into
Conversation
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Review fixes from CodeQL + Gustavo on the native ACE-Step CPU port: - CodeQL (9 sites): widen int*int to int64_t/size_t before the ggml dimension / index conversion (dit_ggml, lm_ggml, qwen3_block, vae_ggml). - Windows portability: new header-only gguf_mmap.h (POSIX mmap / Win32 CreateFileMapping+MapViewOfFile); dit_gguf/vae_gguf now use it instead of raw POSIX open/mmap/munmap/close. - Tests: new test/test_acestep_units.cpp wired into ctest (LABEL "unit", no GGUF fixture) covering dit_build_schedule, philox_randn (torch.randn parity golden), fsq_decode_index and sample_top_k_p. - Metadata: engine now returns the Phase-1-enriched prompt.* (bpm, keyscale, vocal_language, caption) and parses timesignature to the int field, instead of echoing the raw params.*. - Denoising tail: pad frames [T_25Hz, T) are filled with the silence latent and keep chunk mask = 1.0, matching acestep.cpp (not zeroed). - MetadataFSM::init log gated behind a verbose flag threaded from the LM params (production addon path runs quiet). - resolve_paths: anchor on ACE-Step filename stems instead of the bare "lm" substring. - bpe_decode: cache the byte->char reverse map on the tokenizer at load instead of rebuilding it per call. - Remove QVAC ticket references from source comments; drop the plan docs.
Move the ACE-Step music-generation engine out of tts-cpp into a new standalone audiogen-cpp/ engine folder (sibling of tts-cpp / parakeet-cpp), addressing the PR #98 review. tts-cpp no longer builds any acestep sources. - git mv src/acestep/* + include/tts-cpp/acestep/{engine,vae}.h + the unit test into audiogen-cpp/ (history preserved as renames) - new standalone CMake project: find_package(ggml) via the ggml-speech fork (needs the custom snake / col2im_1d ops), own export.h (AUDIOGEN_API) + audiogen-cppConfig; builds the lib, the CLIs/smoke harnesses and the CPU-only unit test (ctest -L unit) - drop the tts-cpp backend_selection dependency (CPU-only, uses ggml_backend_cpu_init directly) - remove all acestep sources / executables / test from tts-cpp/CMakeLists.txt C++ namespace stays tts_cpp::acestep (minimal churn). Verified: standalone audiogen-cpp build is green and the unit test passes; tts-cpp reconfigures clean without acestep.
Add optional on_step hooks so the engine can surface real generation
progress to its ProgressFn, driving a determinate UI progress bar
instead of an indeterminate spinner.
- dit_ggml: DitSampleParams.on_step(step, num_steps), fired at the start
of each Euler step; returning false requests cooperative cancellation.
- lm_pipeline: LmSampleParams.on_step(cur, max_tokens), fired throttled
(every 8 tokens) in the Phase-2 code loop. This is the longest stage on
CPU, so it drives most of the bar. Informational only (no cancel).
- engine: wire both hooks to the existing ProgressFn as report("lm", ...)
and report("dit", ...).
No public API change: only the internal DitSampleParams/LmSampleParams
gain a default-empty callback; engine.h's ProgressFn is reused as-is.
b61530e to
d27c3eb
Compare
The VAE decode was a single opaque ggml_backend_graph_compute call, so the progress bar froze on the last DiT step until the whole render finished. Run the decode through a ggml_backend_sched with an eval callback that fires once per computed node, turning the VAE into fine-grained (done/total) progress. - vae.h / vae_api.cpp: Vae::decode takes an optional ProgressCb(done,total); returning false cancels the decode (empty result). Throttled to ~1% steps. - vae_ggml.cpp: replace gallocr + graph_compute with ggml_backend_sched. The scheduler requires a CPU backend as the mandatory fallback, so pass [GPU, CPU] when running on a GPU and [CPU] when already on CPU; with op_offload=false and every VAE op supported on the GPU (snake / col2im_1d Metal kernels) nothing actually falls back, so results are numerically identical. - engine.cpp: wire the VAE callback into the engine ProgressFn as the "vae" stage and honor mid-decode cancellation.
| sp.on_step = [&](int step, int total) { return report("dit", step, total); }; | ||
|
|
||
| std::vector<float> latent; | ||
| if (!dit_sample(m->dit, sp, latent)) throw std::runtime_error("acestep engine: DiT sample failed"); |
There was a problem hiding this comment.
The new per-step cancellation added in this PR is unsafe here. dit_sample returns false both on a real failure and when on_step/report("dit", ...) returns false for a cooperative cancel. In dit_sample, if (p.on_step && !p.on_step(step, p.num_steps)) return false; is indistinguishable from a compute failure, so cancelling during the DiT stage (the long pole, where a user is most likely to cancel) throws runtime_error instead of returning empty PCM. This contradicts the engine contract in engine.h ("Generate music from a text prompt. Empty pcm on cancellation.") and is inconsistent with the LM and VAE stages, which cancel gracefully. If the addon ever passes a cancelling ProgressFn, the exception crosses the addon boundary. Distinguish cancel from failure (e.g. return the cancel state via an out-param or check cancel_flag after dit_sample returns false and return result; instead of throwing).
| sp.real_enc_S = &enc_S; | ||
| // Surface per-step diffusion progress (the long pole) to the caller's | ||
| // ProgressFn; returning false here also honours cooperative cancellation. | ||
| sp.on_step = [&](int step, int total) { return report("dit", step, total); }; |
There was a problem hiding this comment.
The progress/cancellation plumbing added by the two newest commits (report, dit_sample on_step cancel, lm_pipeline throttled on_step, vae_eval_cb node-count throttling + mid-decode cancel) is behavior that ships with no test. Per coding-standards.mdc ("Test every behavior change") and qvac-whisper-cpp.mdc ("Add tests for every ... change"), add a unit test that exercises cancellation (a ProgressFn returning false at each stage yields empty PCM, no throw) and the vae_eval_cb throttle logic. The existing test_acestep_units.cpp covers only the weight-free math (schedule/philox/fsq/sampler), not the new callback logic.
|
|
||
| // VAE tiling for bounded memory on long tracks (see vae.h decode path). | ||
| int vae_chunk = 1024; | ||
| int vae_overlap = 64; |
There was a problem hiding this comment.
vae_chunk and vae_overlap are never read: Engine::generate calls m->vae->decode(latent, T, ...) in one shot and Vae::decode has no tiling. The comment points at a "vae.h decode path" that doesn't tile, so the fields are dead config and the comment is misleading. Either wire the tiling (the vae.h docs warn memory scales with decoded length, so long durations will allocate the full graph) or drop the two fields and the comment.
| auto * p = static_cast<VaeNodeProg *>(ud); | ||
| p->done++; | ||
| if (p->cb && *p->cb) { | ||
| int pct = p->total > 0 ? (int) ((long long) p->done * 100 / p->total) : 0; |
There was a problem hiding this comment.
total is ggml_graph_n_nodes(gf), but the eval callback fires per node the scheduler actually computes; with a GPU+CPU backend list the scheduler can insert copy/split nodes, so done can exceed total and the reported percentage can go past 100. Cosmetic only (progress bar overshoot), but clamp done/pct to total/100 to keep the reported progress monotone and bounded.
VAE: decode long latents in overlapping time windows (core 256 / overlap 48
latent frames) and stitch the audio. The Oobleck decoder allocates a single
arena that grows linearly with T_latent; for a full song it exceeds Metal's
max buffer length on iOS and ggml_gallocr returns a null buffer -> segfault.
Windowing keeps the peak arena ~1.2 GB while staying numerically identical to
the non-chunked path for short latents. Per-node progress is mapped onto a
single 0..100% VAE bar across chunks.
engine: when a GPU is active, default the autoregressive LM to the CPU backend.
On iOS A-series Metal the LM yields empty/garbage logits ("LM produced no audio
codes") while the same weights decode correctly on CPU; the one-shot text/cond
encoders + DiT + VAE stay on the GPU. Env escape hatches (no rebuild):
ACESTEP_LM_GPU=1 forces the LM back to GPU (desktop bench), ACESTEP_ENCODERS_CPU=1
moves the encoders to CPU. Also free textenc/LM/cond after their stage in
low-mem mode and lazily reload next generation to cut peak memory on device.
4 participants
Summary
Adds a native C++ ACE-Step 1.5 music-generation pipeline to
tts-cpp(
tts_cpp::acestep::Engine), running fully on CPU via ggml. No Python, nosubprocess — the whole pipeline is portable C++ for cross-platform builds.
Pipeline: Qwen3-Embedding text encoder → ACE-LM (metadata Phase 1 + audio
codes Phase 2) → FSQ detokenizer → cond/text encoders → DiT (flow-matching)
→ Oobleck VAE → stereo 48 kHz PCM.
Merge order (important)
This PR depends on custom ggml ops that live in a separate PR:
qvac-ext-ggml(addsggml_snake+ggml_col2im_1d,used by the Oobleck VAE decoder) and bump ggml.
What's included
tts-cpp/src/acestep/— full pipeline: LM (lm_ggml,lm_pipeline),BPE tokenizer, metadata FSM, FSQ detok, text/cond encoders, DiT graph +
flow-matching sampler, Oobleck VAE.
torch.randn(CUDA) parity.music-cliend-to-end driver + per-stage smoke tools.CPU-parity fixes (validated vs acestep.cpp --dump, fixed seed)
int64end-to-end; the DiT noise uses the full seedvia Philox (LM sampler keeps a
uint32mt19937 seed by design). Fixesoverflow that decorrelated the noise.
the timbre encoder, so
enc_hiddencarries the timbre token(
[lyric|timbre|text]), matching upstream (194 → 195 tokens).use_timbre_clsis detected from
acestep.encoder_hidden_size > 0.the high-frequency "robotic" artifact is gone.
Review follow-ups (already applied)
dit_ggml.t_r == t, single conditional pass /no CFG); base/sft CFG parity is flagged as not yet verified.
ACESTEP_DUMP_DIR/ACESTEP_INJECT_*) compiledout by default, behind
-DACESTEP_PARITY_DEBUG.verbose; the addonruns with
verbose=false, so production stderr stays quiet.