About fine-tuning for inpainting

[cats-food]

Huge thanks for your code contribution first!

I used your config file "v1-finetune-for-inpainting-laion-iaesthe.yaml" to fine-tune the model for text-conditioned inpainting. The dataset I used is this subset of the Liaon dataset.

It turns out the results finally become the naive inpainting (simply fills the missing region), and were no longer controlled by the text conditioning as the training proceeds (as shown below, the txt prompt is "a cat on the bench", but no cat appears). Maybe i miss some tricks, I wonder did you meet the same issue when you trained the model?

Thank you in advance :)

[franklinosei]

Hello @Yang-Shiyuan,

Have you found a fix for that?
I've also been trying to finetune the inpainting model for a specific problem I have but I can't seem to get the setup right.
Do you mind sharing your training script with me? Just to help me get started. Thanks

[zaryabmakram]

Hey @Yang-Shiyuan,

Can you describe the exact steps you took to achieve these results in a bit more detail? It would be really helpful to know the specifics of your training setup.