Dreambooth LoRA finetune on a trained model (pth to diffusers)

[yayueLaurazhou]

I trained a Efficient-Large-Model/Sana_1600M_1024px/checkpoints/Sana_1600M_1024px.pth to generate image in a certain style(camera angle and composition), the checkpoints are in .pth

I want to use this script below to dreambooth LoRA finetune the model I trained to generate a certain object,

accelerate launch --num_processes 8 --main_process_port 29500 --gpu_ids 0,1,2,3
train_scripts/train_dreambooth_lora_sana.py
--pretrained_model_name_or_path=$MODEL_NAME
--instance_data_dir=$INSTANCE_DIR
--output_dir=$OUTPUT_DIR
--mixed_precision="bf16"
--instance_prompt="a photo of sks dog"
--resolution=1024
--train_batch_size=1
--gradient_accumulation_steps=4
--use_8bit_adam
--learning_rate=1e-4
--report_to="wandb"
--lr_scheduler="constant"
--lr_warmup_steps=0
--max_train_steps=500
--validation_prompt="A photo of sks dog in a pond, yarn art style"
--validation_epochs=25
--seed="0"
--push_to_hub

but it seems like it doesn't support pth models, only diffusers, any solution?

[lawrence-cj]

@yayueLaurazhou This script will definitely help you: https://github.com/NVlabs/Sana/blob/main/tools/convert_sana_to_diffusers.py

BTW, we have prepared the corresponding diffusers version for you: https://huggingface.co/Efficient-Large-Model/Sana_1600M_1024px_diffusers

[yayueLaurazhou]

Thank you for your response！
my trained .pth model works for inference using diffusers library using SanaPipeline

however, when i try to use https://github.com/NVlabs/Sana/blob/main/tools/convert_sana_to_diffusers.py to convert it, it shows the following error

File "/mnt/sdc/zhouyayue/projects/Sana/tools/convert_sana_to_diffusers.py", line 446, in
main(args)
File "/mnt/sdc/zhouyayue/projects/Sana/tools/convert_sana_to_diffusers.py", line 65, in main
all_state_dict = torch.load(file_path, weights_only=True)
File "/home/zhouyayue/miniconda3/envs/sana/lib/python3.10/site-packages/torch/serialization.py", line 1096, in load
raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options
(1) Re-running torch.load with weights_only set to False will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
(2) Alternatively, to load with weights_only=True please check the recommended steps in the following error message.
WeightsUnpickler error: Unsupported global: GLOBAL numpy._core.multiarray._reconstruct was not an allowed global by default. Please use torch.serialization.add_safe_globals([_reconstruct]) to allowlist this global if you trust this class/function.

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.

I tried to turn the weights_only=False, but the diffusers it converted into only generates pure black images, what is wrong with my weights conversion? @lawrence-cj