Describe the bug
Huge memory leak in loss backward when enabled torch activation checkpoint & zero3 & single-model-multi-branch network.
To Reproduce
Steps to reproduce the behavior:
Training any single-model-multi-branch network with torch activation checkpoint & zero3 optimizer, e.g training diffsynth's animate-v2-lora model
- git clone https://github.com/modelscope/diffsynth-studio
- update
examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh add --config_file examples/wanvideo/model_training/full/accelerate_config_zero3.yaml to the launch commad
- running: sh examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.sh
- and the GPU Memory will increase quickly during the loss backward
Expected behavior
Should release the zero3 all-gathered parameters during loss backward even AC is enabled and recompute happened
Screenshots

System info (please complete the following information):
- OS: [Ubuntu 22.04]
- GPU count and types [single machines with x8 32G RTX5090]
- Python version: 3.10.14
Launcher context
accelerate launch with deepspeed zero3 optimizer
Docker context
No Docker context
Additional context
After some debug and text, I got the memory leak is caused by the wrong ds_grads_remaining reset and count
in deepspeed/runtime/zero/parameter_offload.py's PostBackwardFunctionModule hook,
In AC & zero3 & single-model-multi-branch network env, the PostBackwardFunctionModule would be registered double times but the ds_grads_remaining is reset by _run_after_backward_function in every hook register so post_backward_function only run once, but single-model-multi-branch network should run multiple times depends on the number of branches, so the params gathered by pre-hook would never be released.
Describe the bug
Huge memory leak in loss backward when enabled torch activation checkpoint & zero3 & single-model-multi-branch network.
To Reproduce
Steps to reproduce the behavior:
Training any single-model-multi-branch network with torch activation checkpoint & zero3 optimizer, e.g training diffsynth's animate-v2-lora model
examples/wanvideo/model_training/lora/Wan2.2-Animate-2-14B.shadd--config_file examples/wanvideo/model_training/full/accelerate_config_zero3.yamlto the launch commadExpected behavior
Should release the zero3 all-gathered parameters during loss backward even AC is enabled and recompute happened
Screenshots

System info (please complete the following information):
Launcher context
accelerate launch with deepspeed zero3 optimizer
Docker context
No Docker context
Additional context
After some debug and text, I got the memory leak is caused by the wrong
ds_grads_remainingreset and countin
deepspeed/runtime/zero/parameter_offload.py'sPostBackwardFunctionModulehook,In
AC & zero3 & single-model-multi-branch networkenv, thePostBackwardFunctionModulewould be registered double times but theds_grads_remainingis reset by_run_after_backward_functionin every hook register sopost_backward_functiononly run once, butsingle-model-multi-branch networkshould run multiple times depends on the number of branches, so the params gathered by pre-hook would never be released.