-
Notifications
You must be signed in to change notification settings - Fork 253
Add enable_profiling in Runtime Options
#1949
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
| choco uninstall llvm --yes | ||
| python -m pip install "numpy<2" coloredlogs flatbuffers packaging protobuf sympy pytest | ||
| python -m pip install onnxruntime-qnn | ||
| python -m pip install onnxruntime-qnn==1.25.0.dev20260126001 -i https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to specify a version here or would not listing a version and using the latest nightly package be sufficient?
| displayName: 'OnnxRuntime version' | ||
| type: string | ||
| default: '1.23.0' | ||
| default: '1.25.0-dev-20260125-1205-727db0d3dc' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are default values for publishing the official packages for ORT GenAI. I think we should keep the defaults as a stable version of ORT. We can always override what version of ORT to use when the packages are built.
| } else if (strcmp(value, "1") == 0) { | ||
| run_options_->EnableProfiling(ORT_TSTR("onnxruntime_run_profile")); | ||
| } else { | ||
| auto ToProfileString = [](const char* s) -> std::basic_string<ORTCHAR_T> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Why is converting a
char*tobasic_string<ORTCHAR_T>before going back to achar*needed? - Can you add a comment here to explain that this
elsecondition is for a custom prefix for the log file? - Can the
else ifandelseconditions be merged since they both enable profiling?
Background
Previously, developers could only enable profiling through genai_config.json, which profiles the entire lifetime of a session—from session creation to session destruction. When the number of runs is large (for example, generating ~3000 tokens during inference), the profiling output can become excessively large, causing profiling to fail due to oversized files.
Description
This PR addresses this limitation by introducing run-level profiling. We expose
enable_profilinginRuntimeOptions. Developers can now enable profiling for specific runs only.With the following code, two profiling JSON files will be generated using the
run_profiler_fileprefix:• one containing profiling data for the 100th run
• another containing profiling data for the 101th run