WAN training - The rules of the trade #182
Replies: 10 comments 17 replies
-
|
One thing we know is that fp16 precision is generally better than bf16 for Wan: https://blog.comfy.org/p/updates-for-wan-21-and-hunyuan-image so you might consider switching to that. Also you should not need fp8_t5 on a 4090 I don't need it on a 4070 TS with 16GB(though you'd need the full size model ofc.) Wan, like Hunyuan also seems to benefit from LoraPlus("--network_args loraplus_lr_ratio=X" where X is the multiplier for the LR on lora_b blocks, 2 or 4 seems good for Wan) That's all I can tell you about Wan I've only just begun with it! I'm glad we have a discussion place now! Oh one more thing: --fp8_scaled can be used in combination with --fp8_base to employ a scaling algorithm that kohya created/ported from one based on HyVideo. It allows the model to maintain a higher accuracy in fp8 precision by being more thoughtful about the conversion back and forth basically. fp16 base model plus --fp8_base and --fp8_scaled is likely the way to go in my experiments. You can inference with scaled too, it's implemented for wan_generate_video.py and I've ported it to WanVideoWrapper (and also HunyuanVideoWrapper which it's a HUGE boon for) also although I think kijai wants to do his own implementation, it took me a sec to get mine working right. |
Beta Was this translation helpful? Give feedback.
-
|
Will try your recommendations out and will post some results/comparison! Thanks for the input! |
Beta Was this translation helpful? Give feedback.
-
Can you tell me how you set it up? My 16g 3070 is not enough to train 5 x 5s @256x256. swap block = 26. |
Beta Was this translation helpful? Give feedback.
-
|
@AndreRatzenberger I had discrete_flow_shift set to 3 also as listed here but then when visiting the repo the other day I saw this note on the main page: https://github.com/kohya-ss/musubi-tuner?tab=readme-ov-file#usage At the time I hadn't realized that was only on the main page and not on the document page where it suggests using the same values for Wan you do Hunyuan. Is 3 definitely the value that we should use for training Wan i2v (not 5 or 7?) Thanks for any insight!! |
Beta Was this translation helpful? Give feedback.
-
|
hi wnaybody know what is wrong here , it gave me very hard time (hun_env) C:\ai\musubi-tuner>accelerate launch --num_cpu_threads_per_process 1 ^ INFO:dataset.image_video_dataset:bucket: (320, 320, 129), count: 1 epoch 1/1600 (hun_env) C:\ai\musubi-tuner> |
Beta Was this translation helpful? Give feedback.
-
|
Do you intend for the launch commands to end at --Seed 42? You're missing a few things like --output_dir. What likely happened is that editing the file you deleted the ^ that's supposed to be after 42 which would tell the console there are more commands forthcoming. Currently the commands your executing end at --seed. So maybe take a look and confirm that each line (until the final line) has a ^ symbol followed by a carrier return (enter). |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for replying; it made me crazy , any help appreciated Here is my toml file [[datasets]] Training command : accelerate launch --num_cpu_threads_per_process 1 ^ (hun_env) C:\ai\musubi-tuner>accelerate launch --num_cpu_threads_per_process 1 ^ INFO:dataset.image_video_dataset:bucket: (320, 320, 129), count: 1 epoch 1/1600 (hun_env) C:\ai\musubi-tuner> |
Beta Was this translation helpful? Give feedback.
-
|
This is saying you only have 1 video in total for you dataset: INFO:dataset.image_video_dataset:found 1 videos I assume you have more than 1 file in: "dataset/venom". Try writing the path differently. For example for my current dataset.toml I have "g:/musubi-tuner/training_videos" for my dataset, and for my tensorboard logs I have /logs, and THAT (w/ the leading slash) goes to g:\logs on my Windows workstation. So perhaps it's looking at the wrong folder looking for your dataset. Also make sure that there are files in your cache directory which you list as: "dataset/venom/cache" - You need to run two command line processes prior to launching training. They will create the latent/txtenc cache files (should be a bunch of .safetensors in that cache folder if you did it properly). See her for the Pre-Caching commands if you didn't catch that this needs to be done before training: https://github.com/kohya-ss/musubi-tuner/blob/main/docs/wan.md#pre-caching--%E4%BA%8B%E5%89%8D%E3%82%AD%E3%83%A3%E3%83%83%E3%82%B7%E3%83%A5 |
Beta Was this translation helpful? Give feedback.
-
|
@CCpt5 even if you have video still video frames (images) are trained right? not a video stream |
Beta Was this translation helpful? Give feedback.
-
Yes, I think you always have to pre-cache due to VRAM limits. |
Beta Was this translation helpful? Give feedback.
-
|
I use flow shift = 5.0 for Wan, = 7.0 for Hunyuan. I haven't trained for I2V specifically yet. Basic tips for saving VRAM/avoiding OOM:
Advanced: Super Advanced(Linux only): |
Beta Was this translation helpful? Give feedback.
-
|
Very good tips! Training to step 12 with 256x256 video always suggests that oom needs 350M more vram. |
Beta Was this translation helpful? Give feedback.
-
|
I just noticed for samples during training Wan (i2v) Musubi is using a dicrete_flow_shift value of 14.5 which is way higher than the 5 I'm training on. I didn't change that value nor set it specifically in the prompt.txt. Should I change this value to 5 in the hv_train_network.py file: discrete_flow_shift = sample_parameter.get("discrete_flow_shift", 14.5) Or is that value w/ 20steps a fair representation of how the LoRA will work later under normal Comfy WFs? Looks like it's set to 14.5 default due to the HY paper? There's a mention to do testing on the inference expample page: But also this about HYV's paper: musubi-tuner/docs/sampling_during_training.md Line 101 in c8fea74 Can someone please advise if I should change this for the most accurate representation of where my training is at? |
Beta Was this translation helpful? Give feedback.
-
I would set --fs to 5.0 when sampling for Wan, 7.0 for sampling with Hunyuan - when using the default step count of 50. If you are using less steps, you should use more flow shift. Higher flow shift basically allows more change to happen in earlier timesteps which you need if you are sampling less of them. |
Beta Was this translation helpful? Give feedback.
-
|
Great thanks! |
Beta Was this translation helpful? Give feedback.
-
|
If you don't already know, the description of this model published on Civitai is very informative: https://civitai.com/models/1404755/studio-ghibli-style-wan21-t2v-14b Many thanks to him. |
Beta Was this translation helpful? Give feedback.
-
DAMN and I thought I did in depth, verbose write ups on this stuff! That's a super great resource and I expressed my thanks to the writer as well! |
Beta Was this translation helpful? Give feedback.
-
|
Some more things I can add now that I have more experience: With Wan I get better results with I've had really good luck using an LR of 2e-5 with a LoraPlus of 4 with Wan, sticking to low resolutions of around 480x272 for video. Converting the video to 16fps beforehand DEFINITELY helps(I've been creating my datasets in 24fps for Hunyuan and then using a script that employs ffmpeg to convert the dataset to 16 for training Wan.) I find Wan notably easier to train than Hunyuan (possibly because Wan uses CFG and Hunyuan has embedded? I had more trouble training Flux with it's embedded too until the CFG De-distilled came out to train on!) It's worth nothing that Wan LoRAs tend to work both for T2V/I2V mode, I think training on T2V is better if you wanna go for both, training on I2V produces better results for I2V but worse for T2V. Let's see what else... still shot samples (1 frame) are more useful when training/motion video than you'd think, so if you don't wanna splurge for a video sample I highly recommend that instead. I think that's all I've got to add! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for coming back and updating! I also find alpha=dim seems to be more effective for me w LR properly adjusted. I've been training more ob the Skyreels models lately (at 24fps) and I have had some good results doing that. Not enough testing to say it's "better" but I've been liking recent results and that the only change I've made in my procedure. |
Beta Was this translation helpful? Give feedback.
-
|
Skyreels V2 is pretty good in my experience(I haven't trained it, but I've tested the base model for inference), it's just a bit heavy for someone like me with 16GB VRAM and 4070 TI Super. The lowest res videos I regularly make are 832x624 and because of the extra frames required for a given length because 24fps plus the lack of embedded guidance(which helps Hunyuan a lot here) and the extra blocks that must be swapped as a result to boot, it just takes too long for my taste. I'm pretty on about quality and I try to shoot for around 20 mins per gen at that res, and I rarely make anything except 5 seconds length, skyreels takes me nearly twice that. And for 960x720 which I do sometimes and can get out of Hunyuan or Wan in ~ 30 mins, Skyreels takes me 50 for the same. Basic gen settings of SageAttn + compile + fp16_accumulation + fast RoPE on my blissful-tuner version of Musubi. FP8 scaled precision. No fp8_fast for Wan/Skyreels but yes for Hunyuan. That said if it's working for you and your setup, yeah the quality is nice the fluidity of motion is very good! I've actually come to see Wan's 16FPS as a bit of a plus because I don't tend to make extreme motion scenes(I focus on intimate moments, emotions, sensual things, sexuality, etc), and I can VFI my output to 32FPS with my scripts and it's plenty smooth, while requiring only 2/3 the frames generated! Edit: I've also experimented with a workflow where I generate the video first with Wan at 832x624 or similar because I get better prompt responsiveness out of it usually, then do a 30% V2V with Hunyuan at 960x720 to top up the overall quality because I think it's outputs generally look nicer. Even though Wan outputs 16 natively and Hunyuan 24, with only 30% denoise it seems to work very well regardless of that! I still have it output at 16 fps even after the V2V, 30% doesn't change it enough to change the effective frame rate. Then I just do 2x VFI! |
Beta Was this translation helpful? Give feedback.
-
|
Ya'll I've been having EXCEPTIONAL results training Wan with the following config: LR 2e-5 with LoraPlus of 4 And for my dataset.toml: Videos preprocessed into ~5 seconds clips showing the subject of interest at 16fps. This has produced such stellar results that I've gone back and retrained all my Wan models with these settings! You can get away with just the low res 480x272 bucket but it will reflect in the quality of the learned material. Including the higher res shorter clips allows showing the detail while including the lower res longer clips allows showing the progression of an action or scene. Also some of you have likely realized this but you can use width x height x frames which I've dubbed "framepixels" as a guide for mentally computing VRAM usage. For instance 480x272x65 is 8,486,400 or ~8.5 megaframepixels. That means any other buckets such as 848x480x21 that are also ~8.5 megaframepixels will use a very similar amount of VRAM! If you know that you can do 480x272x65 with 20 blocks swapped for instance, then you know you can also do the other two with that same amount of blocks since they are the same amount of framepixels or less! Hope it helps! |
Beta Was this translation helpful? Give feedback.
-
|
How much is the iteration speed per step? |
Beta Was this translation helpful? Give feedback.
-
|
Oh I didn't see this question before I apologize! On my setup I get around ~16-20 seconds per iteration on a config like that. That's with my 4070 Ti Super 16GB. My DDR4 Memory is quad channel with very tight timings which helps a lot when swapping lots of blocks and fortunately my mobo's PCIe3 doesn't seem to hold it back! My current run is using my mixed precision transformer(base weight params in fp16 scaled to fp8 with fp8_scaled, "special" params like head, norm, bias kept in fp32 all the way), upcasting of quantization and linear transforms and I've got 36 Wan blocks swapped! Currently averaging 18.08 seconds per iteration with an ETA of ~12 hours on 2400 steps. Flash attention 3 beta. No compile/dynamo when training because I can't get it to play nice. |
Beta Was this translation helpful? Give feedback.
-
No worries. That speed looks good. Thanks for the settings. I am going to train a lora on my system soon. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, has anyone stumbled upon this one: If I train network dim and network alpha = the same, then my lora will be all garbled up. It's just messy noise. If I do the default 32/1 it comes out fine. Any ideas? |
Beta Was this translation helpful? Give feedback.
-
|
Network dim = network alpha disables alpha scaling. Network alpha equals 1 maximizes it. You can read more about what's actually going on under the definition for network alpha in my guide: #288 Nutshell: We take network_alpha divided by network_dim, then take the square root of that result to create scale which is then used to amplify the weights during training and reverse the effect during inference. The net effect is complex and there is a lot of misinformation around and I'm not even sure if my own explanation is fully correct but as you've seen turning the effect off tends to allow larger changes to the base model for better or worse. If you look at my post right above yours, I've actually decided that the best settings for me personally for Wan are with the effect off - dim = alpha. But for Hunyuan it seems to be a bit more beneficial. Also note that some projects don't even support the network_alpha feature at all, like diffusion-pipe for instance. In this case they are functionally doing the same thing as dim=alpha. Edit: All that said, generally not using alpha requires different LR and stuff than using it, so the reason for your result is likely that the alpha=dim LoRA is just burnt. Using alpha scaling tends to require higher LR and not using it tends to require lower! |
Beta Was this translation helpful? Give feedback.
-
|
What is the general consensus for captaining video for Wan2.2 training? Any writing style that is working well or learning lessons? The Wan paper seems to suggest very simple captaining "the woman pours coffee". Sometimes my videos have more then one action that could be described, something like "while the camera pans down, the man fishes with dynamite". I'm wondering if I should be captaining video something similar to this: https://alidocs.dingtalk.com/i/nodes/EpGBa2Lm8aZxe5myC99MelA2WgN7R35y |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Let's kick off this board with some general "rules" and concepts to keep in mind when training WAN LoRAs.
This should cover things like:
I'll start:
You know how sometimes a t2v LoRA works surprisingly well for i2v tasks too — and sometimes it does absolutely nothing?
And then there are times where even an i2v LoRA doesn't work for i2v, even though you're sure you did everything right.
So... has anyone figured out the "rules" behind this?
Here’s my current default config:
dataset.toml
this config maxes out my 4090 and I'm pretty happy with it and its results!
Beta Was this translation helpful? Give feedback.
All reactions