[Training] Any way to profile the training of the model?

[martinkorelic]

Describe the issue

Hi, I was wondering if there are any ways we can profile the training session of the training with details about different operations and memory allocation of the ONNX training model?
Something like this is done for inference, but I am not sure the same steps will work if we initiate a training session.

Is there any other way to get details about the training computation of the ONNX model? Are there any plans to add such a feature?

To reproduce

Create a training session and enable SessionOptions.EnableProfiling().
Observe that nothing gets saved after releasing the session.

Urgency

No response

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

PyTorch Version

Execution Provider

Default CPU

Execution Provider Library Version

No response

[AdamLouly]

Hey @martinkorelic, You can use either Torch Profiler or nsys (nsight systems) for profiling.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.