Skip to content

Add support for custom conditioning image #812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

wenyifancc
Copy link

When training the Flux Fill model, custom conditioning images can help the model better learn specific behavioral concepts, such as object removal. Experiments have shown that the model cannot learn specific behaviors by masking alone, but can only learn the features of the masked subject. However, by custom conditioning images and giving the difference between the before and after images, the model can learn specific behavioral features, with satisfactory results.

Example:
Scenario case: Object removal
Base model:Flux-Fill-dev
Dataset:
https://huggingface.co/datasets/lrzjason/ObjectRemovalAlpha
Dataset folder structure:
1-condlabel.png //Before object removal
1.png //After object removal
1.txt //prompt txt such as rmo

Results after 1700 steps of training:
Original image:
image
Masked image:
image
Use flux fill with lora trained before:
Prompt: rmo
Result image:
image

@Nerogar
Copy link
Owner

Nerogar commented Apr 27, 2025

Just FYI: I think this is a great idea that can be really useful. But I see a small issue with the data loader and I haven't thought of a good solution yet.

I'd like to have a fallback to the original implementation for all images that don't have a custom conditioning image. MGDS has a fallback option that can be used for this, but it requires the modules to be added in the correct order. The module that outputs the loaded image needs to be placed after the cond image generation module. And I think the intermediate modules need to be aware of possible None values.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants