Recommedation to decrease VRAM use by half.

[MeiYi-dev]

NVIDIA/cosmos-framework#11

The above PR makes the model run like a standard DiT, in turn making the TE offloadable to RAM and enabling higher duration videos on low VRAM.

[RyukoMatoiFan]

Thanks! I saw this, just was not sure if it makes sense to continue development. This model had some potential, but without official Comfy support it seems DOA for wide adoption.

[MeiYi-dev]

Thanks! I saw this, just was not sure if it makes sense to continue development. This model had some potential, but without official Comfy support it seems DOA for wide adoption.

I think your implementation is so native that a PR could be made by some AI easily that implements it into core ComfyUI. Your implementation already feels core enough! After that it's just a matter of time someone merges your PR. I can let Kijai know about your PR if you want. Plus I think people have not seen any demos of this model. Same thing happened for Kandinsky 5.0. No one uses it but it's better than Wan2.2 for me. Some post training will make the model even better.

Some demos from the 8B model (Most are first seeds)

Cosmos3-native_00070_.mp4

Cosmos3-native_00069_.mp4

Cosmos3-native_00067_.mp4

Cosmos3-native_00052_.mp4

Cosmos3-native_00047_.mp4

Cosmos3-native_00037_.mp4

Cosmos3-native_00035_.mp4

Cosmos3-native_00012_.mp4

Cosmos3-native_00024_.mp4

Cosmos3-native_00021_.mp4

Cosmos3-native_00014_.mp4

Cosmos3-native_00007_.mp4

Cosmos3-native_00008_.mp4

Cosmos3-native_00011_.mp4

Cosmos3-native_00004_.mp4

[RyukoMatoiFan]

@MeiYi-dev I'll try to add the feature from this PR shortly.
Also I discovered there is a bug in my implementation that made the whole pipeline ignore ComfyUI's dynamic VRAM feature. Once I fix it, it should be easier to generate larger videos without OOMing.

Also I should say I made several attempts of fine-tuning a LoRA for this model, and it learns pretty well, but it probably needs a large scale fine-tune (like Anima did with Cosmos Predict) to fully unleash its potential.

[MeiYi-dev]

@MeiYi-dev I'll try to add the feature from this PR shortly. Also I discovered there is a bug in my implementation that made the whole pipeline ignore ComfyUI's dynamic VRAM feature. Once I fix it, it should be easier to generate larger videos without OOMing.

Also I should say I made several attempts of fine-tuning a LoRA for this model, and it learns pretty well, but it probably needs a large scale fine-tune (like Anima did with Cosmos Predict) to fully unleash its potential.

Yeah I read it in the paper. This model is not post-trained at all. (No SFT/RL/low LR training). The model definitely has a lot of potential.

[MeiYi-dev]

As for the PR implementation in native ComfyUI, I think the weights of the reasoner layers and the generation layers can be kept as seperate files and be loaded seperately, so it works like a standard model in ComfyUI. Generator model loaded from the "Load diffusion model" node and the reasoner model loaded with the "Clip Loader" node within ComfyUI, same with the VAEs.

[MeiYi-dev]

@RyukoMatoiFan I think we should make a PR on official ComfyUI repo? The workflow is mostly native nodes and is coded in all torch and is stable for me.

[MeiYi-dev]

Some 32B T2V demos!

ComfyUI_00007_.mp4

ComfyUI_00005_.mp4

[RyukoMatoiFan]

Closing this issue, since with ComfyUI's dynamic VRAM the transformer already streams weights tensor-by-tensor, so the reasoner tower is never resident during denoising and effectively offloaded without any manual split.
This feature could potentially only help users who do not use dynamic VRAM, but I would recommend against doing it anyway.

[MeiYi-dev]

Closing this issue, since with ComfyUI's dynamic VRAM the transformer already streams weights tensor-by-tensor, so the reasoner tower is never resident during denoising and effectively offloaded without any manual split. This feature could potentially only help users who do not use dynamic VRAM, but I would recommend against doing it anyway.

The problem is, the amount of RAM required will be minimum 64GB in order to run the 64B model, by seperating the reasonor and generator, it will be possible to run the 64B variant with just 16GB VRAM and 32GB RAM.

37% of the users compared to 4% 😅 (https://store.steampowered.com/hwsurvey/Steam-Hardware-Software-Survey-Welcome-to-Steam)

Sidenote: We are testing your nodes on Bonodoco https://discord.gg/tZDPtGbP on #cosmos

[RyukoMatoiFan]

I see. I will reopen it and maybe take a look at this later.