New feature: OnnxPredict algorithm #1488
Conversation
| OS=$(uname -s) | ||
| CONFIG=Release | ||
|
|
||
| if [ "$OS" = "Darwin" ]; then |
There was a problem hiding this comment.
@xaviliz, since we are inside debian_3rdparty, should we remove or move somewhere else the MacOS support?
There was a problem hiding this comment.
Yes, that's true. I kept it for testing purpouses. Let me clean it a bit.
There was a problem hiding this comment.
It has been tested on Linux.
| const char* OnnxPredict::name = "OnnxPredict"; | ||
| const char* OnnxPredict::category = "Machine Learning"; | ||
|
|
||
| const char* OnnxPredict::description = DOC("This algorithm runs a Onnx graph and stores the desired output tensors in a pool.\n" |
There was a problem hiding this comment.
it should be an ONNX model, there is no access to graphs in onnxruntime. It is fixed now.
|
|
||
| // Do not do anything if we did not get a non-empty model name. | ||
| if (_graphFilename.empty()) return; | ||
| cout << "after return" << endl; |
| _env = Ort::Env(ORT_LOGGING_LEVEL_WARNING, "multi_io_inference"); // {"default", "test", "multi_io_inference"} | ||
|
|
||
| // Set graph optimization level - check https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html | ||
| _sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); |
There was a problem hiding this comment.
Since there are different optimization options, I'm wondering if there is a chance that extended optimization doesn't work or affects model performance in some cases. I think this should be turned into a parameter that defaults to extended.
There was a problem hiding this comment.
That's a good point, I am not sure how optimizations could affect the performance. Adding new parameter sounds good to me. So, do you propose to add boolean parameter for each optimization? or just an string to use one of them?
There was a problem hiding this comment.
New optimizationLevel parameter has been added as string with choices: {disable_all, basic, extended, all}, by default extended. Maybe it is nice to add some additional tests, what do you think about comparing outputs for the identity model in all extensions?
| // Set graph optimization level - check https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html | ||
| _sessionOptions.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); | ||
| // To enable model serialization after graph optimization set this | ||
| _sessionOptions.SetOptimizedModelFilePath("optimized_file_path"); |
There was a problem hiding this comment.
I think this is mainly intended for debugging purposes. Can we skip saving the optimized graph for efficiency?
https://onnxruntime.ai/docs/api/c/struct_ort_api.html#ad238e424200c0f1682947a1f342c39ca
There was a problem hiding this comment.
yes, we don't need to store the optimized graph in a model.
| return out; | ||
| } | ||
|
|
||
| void OnnxPredict::reset() { |
There was a problem hiding this comment.
Shouldn't we reset the session and env too?
There was a problem hiding this comment.
That's a good point. I couldn't find a reset method for session and env in the CPP_API like in Tensorflow. But let me try it using std::unique_ptr maybe that could work. however, I am doubting if we should do that after compute(), because if we reset the session at the end of configure(), session.Run() will fail.
| const Pool& poolIn = _poolIn.get(); | ||
| Pool& poolOut = _poolOut.get(); | ||
|
|
||
| std::vector<std::vector<float>> input_datas; // <-- keeps inputs alive |
There was a problem hiding this comment.
input_datas -> input_data?
I think data is already plural
| // Step 2: Convert data to float32 | ||
| input_datas.emplace_back(inputData.size()); | ||
| for (size_t j = 0; j < inputData.size(); ++j) { | ||
| input_datas.back()[j] = static_cast<float>(inputData.data()[j]); |
There was a problem hiding this comment.
Instead of forcing casting data to float, shouldn't we keep it in Real format (that is actually float32 by default) and make sure that the model runs in whatever type Real points to?
There was a problem hiding this comment.
That's true when Real == float but we need fall back to cast when Real == double. So, we should not try to make ONNX Runtime run on “whatever Real points to.” However the redundant cast when Real == float could be avoided.
There was a problem hiding this comment.
Fixed #1488! Essentia::Real is already float32 by default, so no need to cast ;)
| } | ||
|
|
||
| // Step 3: Create ONNX Runtime tensor | ||
| _memoryInfo = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault); |
There was a problem hiding this comment.
Would it be possible to run the models on GPU if available?
There was a problem hiding this comment.
yes, it is 6131728
Maybe it is nice to add some test for these functionalities. I couldn't test them properly yet.
src/python/essentia/standard.py
Outdated
| def _create_essentia_class(name, moduleName = __name__): | ||
| essentia.log.debug(essentia.EPython, 'Creating essentia.standard class: %s' % name) | ||
|
|
||
| # print(f"name: {name}") |
…hen cuda, metal or open_ml are not compiled in onnxruntime library
…ETAL or OPEN_ML providers are found
… level in ORT.
- Declared as a new string parameter with choices: {disable_all,basic,extended,all}, by default “extended”.
- https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html#levels
- Set graph optimization level in ORT Session.
- `test_default_optimization_level()`: Check that the default optimization level is 'extended'. - `test_set_valid_optimization_levels()`: Check that valid optimization levels can be set without errors. - `test_set_invalid_optimization_level()`: Check that invalid optimization levels raise an error.
- CUDA tensors now use Ort::MemoryInfo::CreateCuda to allocate GPU memory. - Metal and CoreML providers continue to use CPU tensors; data is managed internally by the provider. - Execution provider is auto-selected based on availability and _deviceId.
|
Thank you @palonso, all changes/suggestions have been addressed. Before merging it would be nice to:
|
2 participants
New feature: OnnxPredict algorithm
Feature
It makes additional changes in Essentia library to build ONNX Runtime Inferencing library from the source and to implement a new algorithm
OnnxPredictfor running ONNX models (.onnx) with multiple IO.Implementation
onnxruntimedynamic library.OnnxPredictto run ONNX models in Essentia.test_onnxpredict.pyPrerequisites
python >= 3.10cmake >= 3.28Testing
How to Test
Tested in
onnxruntime-v1.22.1:python 3.13.4andcmake 4.0.2python 3.10.18andcmake 4.1.0How to build ONNX Runtime
After installing Essentia dependencies in a virtual environment, install
cmakeThen we can run the building script:
cd packaging/debian_3rdparty bash build_onnx.shHow to build OnnxPredict
In MacOS:
In Linux:
python3 waf configure --fft=KISS --include-algos=OnnxPredict,Windowing,Spectrum,MelBands,UnaryOperator,TriangularBands,FFT,Magnitude,NoiseAdder,RealAccumulator,FileOutputProxy,FrameCutter --static-dependencies --with-onnx --lightweight= --with-python --pkg-config-path /usr/share/pkgconfig --std=c++14 python3 waf -v && python3 waf installHow to unittest