How to use extractors binaries with tensorflow

[regorxxx]

So seeing there is no support to get gaia working, nor binaries to run analyzers, I followed Wo80 instructions and build my own:
wo80#5

After quite a bit of help finally got it working:
https://github.com/regorxxx/essentia/actions

Essentia extractors up to date, contrary to official ones:
https://mtg.github.io/essentia-labs/news/2015/12/22/static-binaries-for-extractors/

BUT how am I supposed to use tensorflow models with them? The binaries, when run, say they suppor tensorflow.

streaming_extractor_music.exe ingest .yaml profiles, but everywhere I looked, only SVM models are supported for 'highlevel' data on it.

Here it is the complete list of extractors.
https://github.com/regorxxx/essentia/blob/a0f621aa490371a8dfc097914af717eb246fbbce/src/examples/CMakeLists.txt#L4

So, for example if I want to calculate mood acoustic like I did with gaia, how I do it with tensorflow + extractors?

[regorxxx]

So essentially, correct me if you find something more... things working are:

streaming_vggish_predict.exe -> with some vgg models

I tested it with mood_acoustic-vggish-audioset-1.pb and works

It fails for mood_electronic-audioset-vggish-1.pb, mood_happy-audioset-vggish-1.pb, gender-audioset-vggish-1.pb, ...

streaming_musicnn_predict.exe -> musiccnn models

But then.. as you say, msd-musicnn-1.pb works fine. But mood_happy-msd-musicnn-1.pb for ex. doesn't work.

streaming_tensorflowpredict -> No idea

standard_tempocnn -> https://essentia.upf.edu/models.html#tempocnn (not tested)

EDIT: testing all vggish, most are not working. I don't really understand what's going on. The cmd output says they load fine and then nothing. My impression is there have not been tests related to this in years, and its a bug. It makes zero sense to have binaries for a single random model and the python files seem to be thought to work with any model in general, as long as it matches the binary.

Which musiccnn models work with streaming_musicnn_predict.exe? Why some work and others not? Where is documentation about all this?

Same for streaming_vggish_predict.exe .

What's streaming_tensorflowpredict for?

[xaviliz]

Thank you for taking some time to build static binaries for Windows and sorry for not supporting Gaia any more.

For essentia-models you have all documentation here: https://essentia.upf.edu/models.html

So, for example if I want to calculate mood acoustic like I did with gaia, how I do it with tensorflow + extractors?

Do you mean examples? https://github.com/MTG/essentia/tree/master/src/examples

There is no cpp extractor for mood now. You can do it in python following the examples/snippets in https://essentia.upf.edu/models.html#mood-acoustic or to create your own cpp extractor.

Which musiccnn models work with streaming_musicnn_predict.exe? Why some work and others not? Where is documentation about all this?

Please could you provide more context about this? At least, which ones are failing and which not and error traces to understand better the issues.

Have you checked the outputLayer paramter of the models? It is hardcoded here.

This output layer name only coincides with the predictions output for musicnn- check here https://essentia.upf.edu/models/feature-extractors/musicnn/msd-musicnn-1.json

Same for streaming_vggish_predict.exe .

How do you run it? Can you provide the command?
Are you providing the correct outputLayer?
model/vggish/embeddings is the correct outputLayer for vggish, check here.

What's streaming_tensorflowpredict for?

It looks a generic example to run MFCC extraction to get melspectrogram to feed a model, probably musicnn. But if you check outputLayer doesn't match with the musicnn output. So, it should be created for another model.

Hope this helps.

[regorxxx]

As seen here:
wo80#5 (comment)

We built Essentia windows binaries with both Tensorflow and Gaia. That links points to model not working:

musicnn:
    fs_loop_ds-musicnn-msd-1
    moods_mirex-musicnn-msd-1
    moods_mirex-musicnn-mtt-1
    urbansound8k-musicnn-msd-1
vggish:
    audioset-vggish-3
    moods_mirex-vggish-audioset-1

Commands, bat files and binaries can be found here:
https://github.com/regorxxx/foobar2000-assets/tree/main/essentia-2.1b6

The docs don't really help at all for finding which models work or which ones are not available for binaries and only work on python.

[xaviliz]

Thank you for sharing your comment and files.

I will check it thoroughly and will try to improve the documentation for those examples.
It is something pending for time ago.

We built Essentia windows binaries with both Tensorflow and Gaia. That links points to model not working:

musicnn and vggish models only run in Tensorflow.
Is this only referring to examples in C++? Have you tried them in python?

Commands, bat files and binaries can be found here:
https://github.com/regorxxx/foobar2000-assets/tree/main/essentia-2.1b6

I have been checking your BAT files but I cannot find those models. Can you point them out?
Thanks in advance.