can't successfully resume training from the checkpoint with train_pixart_lora_hf.py

[jingyiz86]

When I run train_pixart_lora_hf.py using the following command:

accelerate launch --num_processes=2 --main_process_port=36667 train_scripts/train_pixart_lora_hf.py --mixed_precision="fp16"
--pretrained_model_name_or_path=PixArt-alpha/PixArt-XL-2-1024-MS
--dataset_name=dataset/test-blip-captions-en --caption_column="text"
--resolution=1024 --random_flip
--train_batch_size=4
--num_train_epochs=200 --checkpointing_steps=100
--learning_rate=1e-06 --lr_scheduler="constant" --lr_warmup_steps=0
--seed=42
--output_dir="pixart-text_2"
--validation_prompt="cute dragon creature" --report_to="tensorboard"
--gradient_checkpointing --checkpoints_total_limit=10 --validation_epochs=5
--rank=16
--resume_from_checkpoint=pixart-test-model/checkpoint-36200

I encounter the error shown in the image:

[jingyiz86]

Although I could manually modify the step in the accelerator file, this approach is somewhat rigid. Is there a better solution?🥹