[Feature Request] add a memory logging manager

[sayakpaul]

Logging accelerator and CPU memory during training models is a common scenario all practitioners run into. As discussed with @muellerzr over Slack, accelerate should have a utility for that (its location would be here).

peft has a good reference here:
https://github.com/huggingface/peft/blob/ae55fdcc5c4830e0f9fb6e56f16555bafca392de/examples/oft_dreambooth/train_dreambooth.py#L421

Opening it up as a feature request in case anyone's interested in contributing this. This would be a massive help, IMO.

[BenjaminBossan]

To clarify, you would like to see memory usage over time, is that right? I wonder how much could be already covered by the profiler integration.

[sayakpaul]

To clarify, you would like to see memory usage over time, is that right?

Yes. In this case, users probably won't care about this level of granularity (as they can always use the PT profiler if they do) but rather aggregated statistics on a few of the important metrics logged by https://github.com/huggingface/peft/blob/ae55fdcc5c4830e0f9fb6e56f16555bafca392de/examples/oft_dreambooth/train_dreambooth.py#L421.

[BenjaminBossan]

Not sure if the torch profiler can be configured to give the information on that aggregation level. But even if it's possible, it should at least be documented, as many users are probably interested in this higher level view.

[sayakpaul]

Yeah if it can be done, I think having it documented even in accelerate should be enough with a minimal end-to-end example.

[BenjaminBossan]

Pinging @yhna940 who added the profiler, maybe they have an idea.