Fixing multiple GPU Qwen Image Fine tuning training #674
+28
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@kohya-ss good news i tested trained on this (1x gpu) and works perfect no issues |
kohya-ss
reviewed
Oct 25, 2025
There was a problem hiding this comment.
It appears that this file has been unintentionally modified.
src/musubi_tuner/hv_train_network.py
Outdated
| if args.ddp_gradient_as_bucket_view or args.ddp_static_graph | ||
| else None | ||
| DistributedDataParallelKwargs( | ||
| find_unused_parameters=True, |
There was a problem hiding this comment.
According to the PyTorch documentation, specifying find_unused_parameters=True when it is not necessary will slow down the training:
https://docs.pytorch.org/docs/stable/notes/ddp.html#internal-design
Therefore, as with other DDP-related options, it would be preferable to be able to specify it as an argument (for example, --ddp_find_unused_parameters).
src/musubi_tuner/hv_train_network.py
Outdated
Comment on lines
1882
to
1886
| # Ensure DDP is properly configured for models with unused parameters | ||
| if hasattr(transformer, 'module') and hasattr(transformer.module, 'find_unused_parameters'): | ||
| transformer.module.find_unused_parameters = True | ||
| elif hasattr(transformer, 'find_unused_parameters'): | ||
| transformer.find_unused_parameters = True |
There was a problem hiding this comment.
There seems to be no point in overriding find_unused_parameters here, it will already be True if configured correctly.
src/musubi_tuner/hv_train_network.py
Outdated
Comment on lines
2123
to
2124
| unwrapped_transformer = accelerator.unwrap_model(transformer) | ||
| logger.info(f"DiT dtype: {unwrapped_transformer.dtype}, device: {unwrapped_transformer.device}") |
There was a problem hiding this comment.
Defining a new local variable unwrapped_transformer may prevent garbage collection later, it is better to call it directly: accelerator.unwrap_model(transformer).dtype and accelerator.unwrap_model(transformer).device.
…ddp_find_unused_parameters argument to hv_train.py and hv_train_network.py - Remove unnecessary override of find_unused_parameters after accelerator.prepare() - Fix unwrapped_transformer variable to prevent garbage collection issues - Revert unintentional changes to qwen_image_train_network.py - Make DDP configuration consistent across all training scripts
|
@kohya-ss i just tried to make the changes you requested I am ok with any way of fixing thank you |
…ddp_find_unused_parameters argument to hv_train.py and hv_train_network.py - Remove unnecessary override of find_unused_parameters after accelerator.prepare() - Fix unwrapped_transformer variable to prevent garbage collection issues - Keep intentional fix in qwen_image_train_network.py for multi-GPU training - Make DDP configuration consistent across all training scripts
|
Thank you for the update. Ruff's lint check is reporting an error, so please format it using Also, it seems that parts of |
sure i will work on that but have you seen this error? Error 2
|
The modification on |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I am not 100% sure if this is accurate way but after this i was able to train
Haven't tested result of training yet
This fixes issue mentioned here : #672