Extension mechanism for video generation script

[kohya-ss]

Highly experimental and may not be merged.

[Sarania]

Ooh🎶 So we've got a mechanism for sharing the generation data and a superclass with several callbacks for different points in the process to provide a base for extensions to interact with the main program, excellent. This could definitely be useful! If it gets merged, I can definitely rework my latent preview and other stuff into this format.

Edit: There's also some other stuff I could add too, honestly I keep trying to use ComfyUI because of the suite of tools available there but keep coming back to Musubi because I simply get the best results with it by far. That's why I keep extending it XD But I think having an official interface would cause more people to do so as well. I would definitely be your first extension dev!

[kohya-ss]

ComfyUI is great, but a bit heavy for quick experiments. I think an extension mechanism would be helpful for users.

However, I also think it is very challenging to develop the expansion mechanism while ensuring future-proofing. I expect there will be frequent changes to specifications, especially in the beginning. The specifications will be decided as we actually implement the extension.

I would be happy if you could be the first developer!

[Auswolf-debug]

honestly I keep trying to use ComfyUI because of the suite of tools available there but keep coming back to Musubi because I simply get the best results with it by far.

I was thinking the same thing....

[kohya-ss]

honestly I keep trying to use ComfyUI because of the suite of tools available there but keep coming back to Musubi because I simply get the best results with it by far.

I was thinking the same thing....

This is a different approach, but it's great. If you need any changes to the script for this, please let me know and I'll do my best to update.

[Sarania]

honestly I keep trying to use ComfyUI because of the suite of tools available there but keep coming back to Musubi because I simply get the best results with it by far.

I was thinking the same thing....

YO THAT'S SICK. I really hope you share that!

[Auswolf-debug]

This is a different approach, but it's great. If you need any changes to the script for this, please let me know and I'll do my best to update.

All good, shouldn't need any changes.

YO THAT'S SICK. I really hope you share that!

Hopefully have it in comfyui manager within a month. Need to make the inference workflows more user friendly.

[Sarania]

I'm working on converting my latent previewer into a class that will more easily be able to be made into an extension. Now with tiny autoencoder powers! I'll endeavour to make it the first extension when we get things sorted.

@Auswolf-debug that's awesome!

Edit: Much cleaner, nicer, and with support for tinyautoencoder for both wan and hunyuan now!

https://github.com/Sarania/musubi-tuner/blob/add-features/blissful_tuner/latent_preview.py

[Auswolf-debug]

Well I've run into a problem and cant find a fix :(

It only effects Wan2.1 v2v using the fun control models, also seems to be a mismatch between tasks and actual available models. From tasks t2v-1.3B-FC, t2v-14B-FC and i2v-14B-FC but only 2 models available from the linked website.?

Anyway using either 1.3b or the 1.4b version, results are the same.

Not sure if anyone has seen this before, but starting to think its a environment problem.