[blog] Add Demucs ONNX export blog - #410
Conversation
|
Please fix the pre-commit issues first (additional space at line end and missing new line at file end) |
23311ce to
20f3274
Compare
daschuer
left a comment
daschuer
left a comment
There was a problem hiding this comment.
Wow, this is really impressive. Thank you.
I have left some comments to make it more accessible for people not that deep into the AI topic.
| - Operates directly on the raw waveform (real-valued input). | ||
| - Uses temporal convolutions and Transformer blocks to learn how each instrument evolves over time. | ||
|
|
||
| ##### **Time-Frequency Domain Branch** |
There was a problem hiding this comment.
I am missing a paragraph like this (my rookie understanding)
The FFT is used to demangle the row of samples to the original bits of sound. Micro oscillations (sinusoid) like emitted of a certain bras sheet of a trumpet. The sum of these sinusoids can be converted back to the streams of sound samples.
The Demux model takes all these resinoids and compares it to the training data and decide which of them belongs to which instrument. This way it can create one stream for each instrument.
There was a problem hiding this comment.
In this blog, I've tried to not talk much about the internal DL architecture, instead focusing only on the export related issues with Demucs. A discussion on the mechanisms of Demucs's layers might be too complex (and too much information) for this blog.
I've referred to convolution and Transformer layers, whose internal mechanisms are well described mathematically but the math doesn't have an easy interpretation. Hence why the AI models are referred to as black boxes.
I can link to the actual research paper though, if you suggest that'd be helpful - https://arxiv.org/pdf/2211.08553
There was a problem hiding this comment.
Your new version reads already good. Thank you.
|
|
||
| #### Real-Valued STFT and ISTFT Rewrites | ||
|
|
||
| Because complex values appear **only** in the initial STFT and final ISTFT layers, our solution was to **reimplement these operations** in a real-valued form. |
There was a problem hiding this comment.
This does not sound correct. Isn't the sine/coin notation just a different notation of the same complex value?
There was a problem hiding this comment.
I think it is correct and uses the common mathematical terms. You could use the polar representation (magnitude and phase) instead of real-value + imaginary value. But to explain this work the later fits better.
sine/cosine are operations - not a representation.
Maybe a simple drawing like https://en.wikipedia.org/wiki/Complex_number#/media/File:Complex_number_illustration_modarg.svg would help the reader to understand the difference between magnitude, amplitude and phase?
There was a problem hiding this comment.
I don't struggling to understand the complex number itself.
For my understanding the sentences is misleading, because the complex-valued nature of the operation must still exists conceptually. It looks like the rewritten STFT avoids using complex tensors, by using a sinus and a cosinus temsor pair that is representing the same info.
z=a+bi=reiθ=r(cosθ+isinθ)
Is that understanding correct?
There was a problem hiding this comment.
I see. Yeah the underlying math is same, but the implementation performs the operations separately for real part and imaginary part, so avoiding any complex tensor data type.
Would you think this statement might be a better fit?
Because complex tensors appear **only** in the initial STFT and final ISTFT layers, our solution was to **reimplement these operations** using only real tensors.
There was a problem hiding this comment.
Your explanation is very good, thank you. Unfortunately it is not integrated in the proposal. Here a try:
| Because complex values appear **only** in the initial STFT and final ISTFT layers, our solution was to **reimplement these operations** in a real-valued form. | |
| Because complex tensors appear **only** in the initial STFT and final ISTFT layers, our solution was to **reimplement these operations** using separate real tensors for the real and the imagine part of the original complex values, while keeping the underlying math unchanged. |
| - The original ISTFT combines complex values through real + imaginary synthesis. | ||
| - We reconstructed the time-domain signal by performing the same series of **overlap-add** and **cosine/sine inverse convolutions**, again using only real-valued tensors. | ||
|
|
||
| By carefully ensuring numerical equivalence to PyTorch’s implementation, we achieved perfect parity (MSE < 1e-4) between the original and rewritten layers. |
|
|
||
| ## Benchmark Results | ||
|
|
||
| The quality of the model is expected to be equal or slightly worse when exported to ONNX. While there are plenty of ways of measuring the benchmarks models (another blog post incoming), we've chosen to measure our models with `SI-SDR` metric, Scale Invariant Signal To Distortion Ratio, on the MusDB dataset. This is the standard metric on which researchers report their source separation model's performance. |
There was a problem hiding this comment.
Can you add other sentence with example db values? What would be perfect what is expected?
There was a problem hiding this comment.
The numerical values for Demucs are in the table below. I don't think additional numbers will improve the text here.
There was a problem hiding this comment.
I struggle to understand which value is better. Is 0 dB the optimum?
There was a problem hiding this comment.
@daschuer
I can probably add a line here that higher dB values are better.
This page is a great way to get familiarized with the SI-SDR metric - https://source-separation.github.io/tutorial/basics/evaluation.html
This is a table for SDR evaluation of Demucs compared to other models - https://github.com/facebookresearch/demucs?tab=readme-ov-file#comparison-of-accuracy
The evaluation of source separated tracks is an important topic in itself. The above URL I've mentioned talks about evaluation of ML models, but doesn't discuss the topic of quality assessment for exported models, which has multiple potential sources of quality degradation. Which is why I've added the comment for (another blog post incoming) in the text.
There was a problem hiding this comment.
That works for me. Thank you.
Interesting links. You may add another sentence summarisimg your good explanations here.
There was a problem hiding this comment.
Idea: for the subject part you may add two files for compassion python vs. c++.
| math: yes | ||
| comments: yes | ||
|
|
||
| Disclaimer: *This blog post primarily serves as the documentation for the [Google Summer of Code](https://summerofcode.withgoogle.com/programs/2025/projects/lRQpeA7K) 2025 project: "Converting Demucs v4 (Hybrid Transformer) AI model to ONNX format".* |
There was a problem hiding this comment.
It would be great to have an eye catching vision statement at the very beginning. Something like:
Imagine you could load a track and .....
|
This type of AI has nothing to do with Chat-GPT. I can imagine that people do not know this. So maybe you may ad short paragraph about pattern recognition. And a brief compare about language models and Audio models. |
|
impressive! thank you. |
|
This project is amazing. This is real, groundbreaking work being done in Mixxx! I love that most of it is merged already, too :) |
|
@daschuer @JoergAtGithub I've edited in your suggestions in the blog |
|
@dhunstack, I hope you understand that it will take me some time to really understand what & how you did it. |
5ea291f to
b3acee2
Compare
acolombier
left a comment
acolombier
left a comment
There was a problem hiding this comment.
This is looking already very good! Just a small nit, but otherwise I think this is ready to go!
daschuer
left a comment
daschuer
left a comment
There was a problem hiding this comment.
I am really exited about your project after reading this Blog. Thank you. LGTM.
|
@acolombier Can you take care for merging? @dhunstack needs a stable link for his submission before the deadline, tomorrow October 27, 2025 8 PM CET. |
2f9d09c to
2b3af21
Compare
70daf44 to
081f098
Compare
Blog documenting the "Converting Demucs v4 (Hybrid Transformer) AI model to ONNX format" GSOC project. Signed-off-by: Anmol Mishra <anmolmishra1997@gmail.com>
|
Thank you very much for this great contribution! It was a pleasure to work with you on this GSoC project! |
6 participants
Blog documenting the "Converting Demucs v4 (Hybrid Transformer) AI model to ONNX format" GSOC project.
preview