Advice for ComfyUI.

[MeiYi-dev]

I think rather than creating custom nodes, you guys should create a PR on the ComfyUI repository itself. You guys only have to code for the DiT model since Qwen2.5 7B and HunyuanVideo VAE are already suported in ComfyUI. This will enable SDPA support, VAE tiling support, low vram usage support, and integration to EasyCache, TeaCache, and the core/custom ComfyUI features. And it will give a lot of attention to your project.

[MeiYi-dev]

@AlekseyLetunovskiy Just getting your attention, as I think it's important for this project.

[FizzleDorf]

I don't think researcher and developer time should be spent on something comfyorg pays it's developers to do as their job.

[MeiYi-dev]

I don't think researcher and developer time should be spent on something comfyorg pays it's developers to do as their job.

It literally will save a lot of the author's time. The authors currently are coding for the VAE, tiled VAE, TE, TE offloading, SDPA support, model offloading support, samplers, schedulers, CFG, every single thing. If they code for ComfyUI directly they would only need about 30-50 lines of code since TE and VAE are already supported along with the rest of the features which are core to the ComfyUI repo. It will even get much more attention if ComfyUI directly supports it.

[FizzleDorf]

it's a diffusers implementation wrapper, it saves them time to just wrap it and move on so the community can demo it right away. if comfyorg didn't have the resources or the manpower it would make sense to consider it but that just isn't the case.

[kabachuha]

Indeed, reimplementing Comfy's features is quite wasting the time could be spent on building the next model version. They also cannot support custom plugin options and samplers which are popping day by day without spending unreasonable amount of resources.

A single Comfy pull-request can help the progress greatly... Assuming, Comfyanon won't reject it on subjective bias, like it just happened with Hunyuan Image-3

[AlekseyLetunovskiy]

Thanks for the advice. We are considering integration into the core ComfyUI as one of our targets.

Actually, when we started thinking about Comfy inference, we considered two options:

1. The first, as you suggested.
2. The second, to create our own wrapper around the inference code.

For the current release, we decided to go with the second option for several reasons:

• It’s difficult to prepare a PR while the model is still closed and the inference code is under active development. In that case, we’d need to maintain two versions of the code and keep them constantly synchronized.
• The Comfy core currently has around 200 open PRs, so we’re not sure how quickly they could review and merge our model.
• Our current implementation of the Comfy pipeline is quite simple in our opinion – you only need to clone our repository and load the models. We can also be sure that if test.py works, then Comfy works as well.
• We’ve made some modifications to the original encoders. For example, when I used the Hunyuan VAE from the core Comfy, it caused OOM even on an H100. The tiled VAE was still experimental and didn’t work properly for our resolutions.

So, we do plan to integrate into the core Comfy eventually – most likely once the code and models have stabilized.

P.S. @MeiYi-dev I saw your issue on Comfy – thank you!

[FizzleDorf]

@kabachuha, it's been a while. would love to get a bunch of the deforum stuff into sdcpp at some point if I weren't swamped already with building out a framework + gui. glad to see you are still active.

@AlekseyLetunovskiy this explanation makes sense and is honestly the norm for model releases. still glad your team decided to make something at all for the comfy folks so thank you for that

[kabachuha]

@AlekseyLetunovskiy If the models are not yet stabilized, it's a fair explanation. As of the point of the PR queue, sound models and streamlined pull requests are merged almost in realtime. Comfy supports modified encoders, for example, for T5 text encoder there is an option to select the type for almost every model.

More about the VAE. Have you considered using DC-AE (ACE-Step/SANA-like, open code) / DC-Videogen (under review, but with open paper)? They not only accelerate training/inference, but are going to be virtually essential for very long/high res video generation in the future, if the plans are to match Wan2.5 in capabilities. As DC-Videogen's paper suggest, the pretrained models can adapt to DC-AEs with quite low amount of fine-tuning.

@FizzleDorf Thanks for enjoying and not forgetting about Deforum. Wan's camera control contraptions are still no match for it :)
I'm not familiar with stablediffusion-cpp and its philosophy though 😅

[silveroxides]

* The Comfy core currently has around 200 open PRs, so we’re not sure how quickly they could review and merge our model.

Model labs and developers of architectures are always priority when it comes to ComfyUI PR. The amount of open PR in a repository should not be a determining factor unless there is a clear pattern of model implementations getting rejected/ignored which it is not

* Our current implementation of the Comfy pipeline is quite simple in our opinion – you only need to clone our repository and load the models. We can also be sure that if test.py works, then Comfy works as well.

Other than importing the basic ComfyABC and progress bar from ComfyUI, your implementation is far from ideal. It pretty much will cause issues due to not integrating any of the model management or other parts that are already present in Core. This makes your implementation only work in isolation and not as part of ComfyUI backend itself.

* We’ve made some modifications to the original encoders. For example, when I used the Hunyuan VAE from the core Comfy, it caused OOM even on an H100. The tiled VAE was still experimental and didn’t work properly for our resolutions.

I would not be surprised if the OOM was because of not properly making use of the model management of ComfyUI. There are several custom nodes that implement tiled VAE in various ways that make use of ComfyUI own backend

Should probably say who I am too. I wrote the ComfyUI implementation for Chroma which got merged quite quickly in to ComfyUI and when we released Radiance, our pixel space model, then that code got merged within like a week.

[silveroxides]

I don't think researcher and developer time should be spent on something comfyorg pays it's developers to do as their job.

This is simply not true. The comfyorg part that is. Pretty much every new model implementation in ComfyUI has been written either by comfyanonymous himself or the researchers/developers in cooperation with comfyanonymous. There is no dedicated part of comfyorg that is focused on backend model implementation. Majority of comfyorg is centered around the frontend(including the desktop app development), the public communication, security implementation and other administrative parts. Backend is still comfyanonymous own repository that is not controlled by the org.