Regions not working with Anima models

[Sen-sou]

The architecture difference is probably what is causing the regions not work with the Anima model or it is yet to be implemented. The closest what i have found for region mask conditioning is the https://github.com/Haoming02/sd-forge-couple which supports anima. Probably could be referred to for implementation.
I have tried porting the same implementation in comfyui as a custom node (using AI) and it gave mediocre results (probably im not doing it right).

doesnt seem to follow the region prompts or rather it completely ignores it

This is what its supposed to generate or something similar.

[muskdavid661-ai]

yes, the prompt in regional are just ignored by the model

[Sen-sou]

Anima base version release

[muskdavid661-ai]

hey do you run forge neo on colab?

[Acly]

The way to implement it would be to add Anima support behind the existing region nodes: https://github.com/Acly/comfyui-tooling-nodes#regions

Then it just needs to be activated in the plugin here

[muskdavid661-ai]

Will the anima region issue be getting resolved in this update or next?

[Sen-sou]

Can implement something like Forge couple (https://github.com/Haoming02/sd-forge-couple) but its pretty weak and not very consistent because it patches only the cross-attention. I tried making something a bit stronger and consistent by patching both the cross-attention and self-attention (https://github.com/Sen-sou/Comfyui-Anima-Regional-Conditioning) (its currently only in comfyui), it works but still has a lot of issues with composition.
So i dont think it is worth adding it here right now.

[muskdavid661-ai]

i see

[Acly]

Self-attention is probably pretty vital to exchange global information and get a good composition and coherent image. Only patching attention-to-prompts was kind of the idea that allowed regions to work with SDXL, the fact that it's not a super strong separation is a feature.
I haven't tried to Anima version though, if it's too weak that's a problem I guess.

[muskdavid661-ai]

Guess, it will take time since the model's structure is different, still thanks for working on solution

[muskdavid661-ai]

hello, i was wondering if you were able to find a solution for it, as regional feature is best of krita compared to other platforms

[Sen-sou]

I tried training a LLLite regional controlnet but it is still not very consistent (https://huggingface.co/Sen-sou/Anima-LLLite-Regional-Controlnet). I tried asking kohya for some tips, but he hasn't replied, so im currently stuck on what parameters would be the best to train the model on.