How to extract the semantic information from WavLM for Mimi training?

[mindmapper15]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The paper

Question

Hi, thank you for your work on Mimi and for sharing all the details in paper.

We are currently working on re-implementing Mimi to train on non-english speech data.

In the paper, you mentioned that the semantic information is distilled from WavLM Large model
We want to know about how exactly the semantic target features are extracted from WavLM Large.

For example, the SpeechTokenizer used HuBERT Base model as the semantic teacher and experimented for both cases: using the 9th layer output and averaged features of all layers that includes input convolution layer output.

So our main question is how exactly the semantic target features were extracted and used in Mimi. The paper only mentions that WavLM Large was used, but it does not provide further details on whether the output from a specific layer was used or whether the outputs from all layers were averaged.

Did you use the output from a particular layer, the average of all layer outputs, or another specific approach such as a weighted average of all layer outputs? We would really appreciate for any guidance.

Thank you for your time.