[feat] Add Cpp scripts for inference - #7
Open
dhunstack wants to merge 1 commit into
Open
Conversation
acolombier
reviewed
Oct 21, 2025
acolombier
left a comment
acolombier
left a comment
Member
There was a problem hiding this comment.
IMO, we should trim down the C++ code to a minimum (remove edge case and redundant libraries) and move this into an examples/cpp folder instead.
The C++ doesn't add any value to demucs itself, and instead shows how to infer an ORT model (+ load/save an audio file with libnyquist). THe current cppscript is misleading as it sounds like if the C++ could be used fort any demucs development
Member
There was a problem hiding this comment.
What is this used for? I cannot see any reference to demucsonnx::stft and demucsonnx::istft. My understanding is that the stft and istft function are already part of the model, is that intending to swap them with explicit implementation? If yes, how does that work with GPU/NPU devices?
Member
There was a problem hiding this comment.
Could you clarify the usecase for this dependency? Looking at this example, it looks like the ONNX library already contains its own data type to manage tensor, which can create from interleaved 32-bits float audio samples. What is the added benefits?
Comment on lines
+82
to
+103
| // create a struct to hold two float vectors for left and right channels | ||
| Eigen::MatrixXf ret(2, N); | ||
|
|
||
| if (fileData->channelCount == 1) | ||
| { | ||
| // Mono case | ||
| for (std::size_t i = 0; i < N; ++i) | ||
| { | ||
| ret(0, i) = fileData->samples[i]; // left channel | ||
| ret(1, i) = fileData->samples[i]; // right channel | ||
| } | ||
| } | ||
| else | ||
| { | ||
| // Stereo case | ||
| for (std::size_t i = 0; i < N; ++i) | ||
| { | ||
| ret(0, i) = fileData->samples[2 * i]; // left channel | ||
| ret(1, i) = fileData->samples[2 * i + 1]; // right channel | ||
| } | ||
| } | ||
|
|
Member
There was a problem hiding this comment.
This looks suboptimal - Instead, you should create tensor directly from the fileData->samples using Ort::Value::CreateTensor, which is already an interleaved buffer. In case of Mono, you could reorganise the buffer (realloc, iterate to reorganise in place). This should be a good usecase of SIMD to optimise, which I believe it cannot at the moment due to the () operator.
Add Cpp scripts for running inference on ONNX exported Demucs Signed-off-by: Anmol Mishra <anmolmishra1997@gmail.com>
Dzide001
pushed a commit
to Dzide001/Deo-MIx
that referenced
this pull request
Jul 13, 2026
First of several planned stages toward on-demand AI stem separation for any regular track (not just pre-made STEM files) using the same native STEM playback engine already built and working (EngineDeck's [Channel1_Stem1..4] controls, stem_count CO, QmlStemsModel, the full QML pad UI). This stage validates the riskiest, most foundational piece in isolation before anything else is built on top of it: does real HTDemucs ONNX inference actually run correctly inside Mixxx's C++ build on this machine. lib/demucsonnx/ (new, MIT, vendored from github.com/dhunstack/demucs branch onnxrt -- the C++ ONNX Runtime inference code from open PRs mixxxdj/demucs#7/#9, produced during Mixxx's own GSoC 2025 "Demucs to ONNX" project, see mixxxdj/mixxx#15495): demucs.hpp, dsp.hpp/cpp, tensor.hpp, model_apply.cpp, model_inference.cpp -- the model-inference core only, not upstream's CLI/libnyquist dependency (Mixxx doesn't need a third file-I/O library for this). One adaptation from upstream: fixed an #include path assuming a submodule layout this build doesn't use (see README.mixxx.md for the full provenance note). src/stemsep/ (new, GPLv2, isolated standalone test harness -- NOT linked into mixxx-lib/mixxx-qml-lib/mixxx/mixxx-test, so a build break here cannot affect the app): a minimal self-contained PCM WAV reader/writer (wavio.h/cpp -- avoids linking libsndfile's static transitive codec dependencies that mixxx-lib gets "for free" elsewhere but a standalone target doesn't) and mixxx-stemsep-test, a CLI that loads a real .onnx model, runs inference on a real WAV, and optionally diffs output against reference stems via correlation coefficient (empirically the right metric here, not raw sample diff -- Demucs applies a random per-call time-shift for a small quality gain, so independent runs of the same pipeline are never bit-identical, and transient-driven misalignment produces large raw diffs even on correct output). CMakeLists.txt: new AI_STEM_SEPARATION option (default OFF, matching the STEM/default_option pattern), gated on two new cache variables (AI_STEM_SEPARATION_EIGEN3_ROOT / _ONNXRUNTIME_ROOT) pointing at dependencies installed OUTSIDE MIXXX_VCPKG_ROOT -- this repo has no vcpkg manifest (deps come from a prebuilt binary bundle) and neither Eigen3 nor onnxruntime exist in it; fully additive, zero impact unless explicitly opted into. Placed after the SndFile block (not next to STEM above) since add_subdirectory() processes immediately and needs SndFile::sndfile already defined -- moot now since stemsep dropped that dependency, but harmless and still correct. Also works around a real packaging quirk in the official onnxruntime-osx-arm64 release tarball: its own CMake config references an "include/onnxruntime" subdirectory that doesn't exist in the tarball (headers ship flat). Verified: configured and built in a separate build-stemsep/ directory (never touching the existing validated build/ tree), ran end-to-end on a real 3:13 track, correlation against a known-good Python reference was 0.94-0.997 across all 4 stems (drums/bass/other/vocals) -- strong confirmation the vendored inference core is numerically correct, not just "builds and doesn't crash." CPU-only execution provider used throughout (no CoreML) -- a CoreML-provider crash was hit with an unrelated ONNX package on this machine during earlier Python validation, so CoreML compatibility here is left for a later stage. Non-goals for this stage (deferred, tracked in the stage 1 plan doc): STEM-file muxing/writing, a background job scheduler, deck/UI wiring, CoreML acceleration. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Cpp scripts for running inference on ONNX exported Demucs