You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ONNX Runtime tests currently generate models and test inputs dynamically inside the Python/C++ test harness. While this works well for ORT CI, it makes it difficult to reuse these test cases outside of the ORT environment.
emx-onnx-cgen generates C code from ONNX models, and we wanted to validate it against a broad set of real-world operator combinations and edge cases from ONNX Runtime tests. However, the current ORT test setup makes these cases hard to extract and reuse.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
ONNX Runtime tests currently generate models and test inputs dynamically inside the Python/C++ test harness. While this works well for ORT CI, it makes it difficult to reuse these test cases outside of the ORT environment.
We ran into this limitation while working on a project:
https://github.com/emmtrix/emx-onnx-cgen
emx-onnx-cgengenerates C code from ONNX models, and we wanted to validate it against a broad set of real-world operator combinations and edge cases from ONNX Runtime tests. However, the current ORT test setup makes these cases hard to extract and reuse.To address this, we built a dataset that materializes ORT test cases into standalone ONNX artifacts:
https://github.com/emmtrix/emx-ort-test-artifacts
Key idea:
model.onnx, inputs (input_*.pb), outputs (output_*.pb)validation.json)Question:
Would a reusable, versioned dataset of ORT test artifacts be useful for ONNX Runtime development or for downstream users?
Possible directions if this is interesting:
Beta Was this translation helpful? Give feedback.
All reactions