Finetuning PixArtMS_XL_2 Model with Custom Data and Multi-Scale Training

[youngwanLEE]

Hi,
I have tried to finetune the PixArtMS_XL_2 model using custom data (e.g., laion-aesthetic data).
I have been closely following the training instructions outlined in the train.ipynb tutorial, especially those related to extracting t5 & vae features from the Pokemon dataset without using the --multi_scale option.

Initially, my data folder structure was designed to follow the tutorial’s guidance, which works well for models like PixArt_XL_2 that utilize the InternalData data class.

However, my goal is to finetune the PixArtMS_XL_2 model, which requires the use of the InternalDataMS data class for multi-scale training. Upon attempting to transition to this setup, I realized that my current data structure is incompatible with the requirements of InternalDataMS.py, which seems to necessitate a different data path configuration (e.g., partition_filter).

It would be better to provide a tutorial for fine-tuning PixArtMS-XL-2-1024-MS model with multi-scale training.

Thanks in advance.

[heyalexchoi]

I ended up forking the repo to load my data. Also made changes to tie together feature paths between extraction and training. And Changes to speed up T5 feature extraction 20X. Lots of upside for contributions here. Unfortunately my PR never got feedback so I closed it. PixArt team may be busy pushing to release sigma rn.

[youngwanLEE]

@heyalexchoi oh good, I will try your PR. Thanks

[lawrence-cj]

I'm sorry for your waiting and thank you for your contribution. I'll re-open your PR and review it. will never let your effort meaningless.

[lawrence-cj]

We release the sigma repo, which reformats the data structure to make it much easier to start your training. You will have totally better feelings. I'm really hoping you can have a try.

[heyalexchoi]

No problem. Can hold off review if training will be moving to Sigma. I also intend to implement validation loss and CMMD measurement for training. However I may end up simply forking your repos depending on how you want to develop in the future. I have found that my changes tend to refactor a bit and change behavior at times. Of course, more than happy to contribute - let me know.

…

On Mon, Apr 1, 2024 at 10:02 AM Junsong Chen ***@***.***> wrote: I'm sorry for your waiting and thank you for your contribution. I'll re-open your PR and review it. will never let your effort meaningless. — Reply to this email directly, view it on GitHub <#102 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABF3GFZ4JFRCGVALK2I46BTY3FSFZAVCNFSM6AAAAABEXESOUCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DSNZTGE4TI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[lawrence-cj]

No problem. If you find that your training is more stable or getting better results, pls let me know, I'd like to try it out.