How generate a contxt model dump from onnx runtime? (C++) #23153
Description
Describe the issue
Situation:
I am loading an onnx model (yolo v5) with TensorRT provider, which takes 4 Minutes on a Jetson Orin.
I successfully speeded this up by caching the TensorRT engine.
OrtTensorRTProviderOptions trt_options{};
trt_options.device_id = 0;
trt_options.trt_max_workspace_size = 2147483648;
//trt_options.trt_max_partition_iterations = 10;
trt_options.trt_min_subgraph_size = 1;
trt_options.trt_fp16_enable = 0;
trt_options.trt_int8_enable = 0;
//trt_options.trt_int8_use_native_calibration_table = 1;
trt_options.trt_engine_cache_enable = 1;
//trt_options.trt_dump_ep_context_model = 1; // desired but not available in trt_options, only in trt2
trt_options.trt_engine_cache_path = "./cache";
//trt_options.trt_dump_subgraphs = 1;
session_options.AppendExecutionProvider_TensorRT(trt_options); // add trt-options to session-options!
But I am not certain about the model security when saving the engine (we are currently loading the model from RAM, so no files are exposed to the a user who has access to the system). Is the trt engine secure, or could anyone generate inference just from that engine file? Especially: Are the model weights inside of the engine, or is the engine just some kind of "meta data" only works in combination with the model file itself (both in ONNX and native TensorRT and some theoretical custom inference engines)?
That's why I would like to embed the engine to a onnx file (context model) and load that model from RAM as before.
If I understand correctly, that should be possible?
For that I add another Provider in addition to the OrtTensorRTProviderOptions trt_options:
std::vector<const char*> option_keys2 = {
"trt_engine_cache_enable"
,"trt_dump_ep_context_model"
,"trt_ep_context_file_path"
,"ep_context_enable"
,"ep_context_file_path"
,"trt_ep_context_embed_mode"
,"trt_engine_cache_path"
//,"trt_timing_cache_enable"
//,"trt_timing_cache_path"
};
std::vector<const char*> option_values2 = {
"1"
,"1"
,"/path1" // sub-path, according to https://app.semanticdiff.com/gh/microsoft/onnxruntime/pull/19154/overview
,"1"
,"/path2" // base path, according to https://app.semanticdiff.com/gh/microsoft/onnxruntime/pull/19154/overview
,"1"
,"/path3"
//,"1"
//,"/path4"
};
Ort::ThrowOnError(api.CreateTensorRTProviderOptions(&tensorrt2_options));
Ort::ThrowOnError(api.UpdateTensorRTProviderOptions(tensorrt2_options, option_keys2.data(), option_values2.data(), option_keys2.size()));
session_options.AppendExecutionProvider_TensorRT_V2(*tensorrt2_options); // add trt2-options to session-options!
However, there I am getting a
[ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:146 Parse Unknown provider option: "trt_ep_context_embed_mode".
How to do it correctly?
To reproduce
Urgency
No response
Platform
Other / Unknown
OS Version
Jetson Orin Linux
ONNX Runtime Installation
Other / Unknown
ONNX Runtime Version or Commit ID
11.4
ONNX Runtime API
C++
Architecture
X64
Execution Provider
TensorRT
Execution Provider Library Version
No response