training using HF LORA - it trains only the transformer, should the text encode be trained as well?

[raulc0399]

checking the code for the HF SDXL Lora script line 1034
i noticed that the text encoder is also trained.

any reason why is not trained for Pixart? i could try it, unless there is a good reason not to? :)

thanks.

[lawrence-cj]

We haven't tried to train the T5. You can give it a try. There is no specific reason for not training. Looking forward for your result~

[lawrence-cj]

Cool. Nice done!

[raulc0399]

added some prints to highlight that training in fp16 can lead to NaNs in loss. also some links with more info.

first tests on fp32 and bf16 however work, subjectively - i think the results are also slightly better :)

[raulc0399]

@lawrence-cj i have also added to the PR the option to train the text encoder at a different learning rate and also to stop training it at a percentage of the total steps

[raulc0399]

@lawrence-cj i have also added a PR for pixart-sigma with changes to train the text encoder

[lawrence-cj]

Cool cool. I will review it asap