fine tuning using HF lora - but on inference it generates as if no peft model is loaded

[raulc0399]

Hi,

i am trying to fine-tune the 512 model using this accelerate launch script:

accelerate launch --num_processes=1 --main_process_port=36667 train_scripts/train_pixart_lora_hf.py --mixed_precision="fp16" \
  --pretrained_model_name_or_path=PixArt-alpha/PixArt-XL-2-512x512 \
  --train_data_dir="../data/train/" --caption_column="caption" \
  --resolution=512 --random_flip \
  --train_batch_size=2 --gradient_accumulation_steps=4 \
  --num_train_epochs=50 --checkpointing_steps=100 \
  --learning_rate=1e-06 --lr_scheduler="constant" --lr_warmup_steps=50 \
  --seed=42 \
  --output_dir="pixart-simpsons-model" \
  --report_to="tensorboard" \
  --gradient_checkpointing --checkpoints_total_limit=10 \
  --validation_epochs=1000 \
  --validation_prompt="cute dragon creature" \
  --rank=16 \
  --use_8bit_adam \
  --dataloader_num_workers=4

the dataset is about 800 imgs with captions.

the inference code is this:

transformer = Transformer2DModel.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", subfolder="transformer", torch_dtype=torch.float16)
transformer = PeftModel.from_pretrained(transformer, "./pixart-simpsons-model")
pipe = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", transformer=transformer, torch_dtype=torch.float16)
pipe.to("cuda")

note: i changed train_scripts/train_pixart_lora_hf.py, since i was getting the error "Attempting to unscale FP16 gradients". so i added this function, from the SDXL training script, to convert the trainable params to float32:

def cast_training_params(model: Union[torch.nn.Module, List[torch.nn.Module]], dtype=torch.float32):
        if not isinstance(model, list):
            model = [model]
        for m in model:
            for param in m.parameters():
                # only upcast trainable parameters into fp32
                if param.requires_grad:
                    param.data = param.to(dtype)

another note:
if i change this in the inference code, as taken from the training script, it generated black images or garbage

transformer = Transformer2DModel.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", subfolder="transformer")
transformer.to("cuda", dtype=torch.float16)
transformer = PeftModel.from_pretrained(transformer, "./pixart-simpsons-model")
pipe = PixArtAlphaPipeline.from_pretrained("PixArt-alpha/PixArt-XL-2-512x512", transformer=transformer, torch_dtype=torch.float16)
pipe.to("cuda")

[lawrence-cj]

How about your training log?

[raulc0399]

i changed the launch script to the one below. it gives better results, but the faces are not right, as well as other details.
below also the training log

accelerate launch --num_processes=1 --main_process_port=36667 PixArt-alpha/train_scripts/train_pixart_lora_hf.py --mixed_precision="fp16" \
  --pretrained_model_name_or_path=PixArt-alpha/PixArt-XL-2-512x512 \
  --train_data_dir="../data/train/" --caption_column="orig_text" \
  --resolution=512 --random_flip \
  --train_batch_size=2 --gradient_accumulation_steps=4 \
  --num_train_epochs=50 --checkpointing_steps=100 \
  --learning_rate=1e-05 --lr_scheduler="constant" --lr_warmup_steps=100 \
  --seed=42 \
  --output_dir="pixart-simpsons-model" \
  --report_to="tensorboard" \
  --gradient_checkpointing --checkpoints_total_limit=10 \
  --validation_epochs=1000 \
  --validation_prompt="cute dragon creature" \
  --rank=4 \
  --adam_weight_decay=0.03 --adam_epsilon=1e-10 \
  --dataloader_num_workers=4

Training log:

04/11/2024 16:53:27 - INFO - __main__ - Distributed environment: DistributedType.NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: fp16

{'clip_sample', 'clip_sample_range', 'rescale_betas_zero_snr'} was not found in config. Values will be initialized to default values.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 9703.42it/s]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:12<00:00,  3.10s/it]
{'latents_std', 'latents_mean'} was not found in config. Values will be initialized to default values.
{'interpolation_scale'} was not found in config. Values will be initialized to default values.
Some weights of the model checkpoint were not used when initializing Transformer2DModel:
 ['caption_projection.y_embedding']
trainable params: 3,441,344 || all params: 614,297,440 || trainable%: 0.5602080972370649
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 756/756 [00:00<00:00, 52073.21it/s]
04/11/2024 16:53:58 - INFO - __main__ - ***** Running training *****
04/11/2024 16:53:58 - INFO - __main__ -   Num examples = 755
04/11/2024 16:53:58 - INFO - __main__ -   Num Epochs = 50
04/11/2024 16:53:58 - INFO - __main__ -   Instantaneous batch size per device = 2
04/11/2024 16:53:58 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 8
04/11/2024 16:53:59 - INFO - __main__ -   Gradient Accumulation steps = 4
04/11/2024 16:53:59 - INFO - __main__ -   Total optimization steps = 4750
Steps:   2%|█▉                                                                                         | 100/4750 [05:42<4:26:01,  3.43s/it, lr=1e-5, step_loss=0.000143]04/11/2024 16:59:41 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-100
Model weights saved in pixart-simpsons-model/checkpoint-100/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 16:59:42 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-100/optimizer.bin
04/11/2024 16:59:42 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-100/scheduler.bin
04/11/2024 16:59:42 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-100/sampler.bin
04/11/2024 16:59:42 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-100/scaler.pt
04/11/2024 16:59:42 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-100/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-100/pytorch_lora_weights.safetensors
04/11/2024 16:59:42 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-100
Steps:   4%|███▉                                                                                         | 200/4750 [11:05<4:02:29,  3.20s/it, lr=1e-5, step_loss=0.0575]04/11/2024 17:05:05 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-200
Model weights saved in pixart-simpsons-model/checkpoint-200/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:05:05 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-200/optimizer.bin
04/11/2024 17:05:05 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-200/scheduler.bin
04/11/2024 17:05:05 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-200/sampler.bin
04/11/2024 17:05:05 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-200/scaler.pt
04/11/2024 17:05:05 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-200/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-200/pytorch_lora_weights.safetensors
04/11/2024 17:05:05 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-200
Steps:   6%|█████▊                                                                                       | 300/4750 [16:29<3:59:08,  3.22s/it, lr=1e-5, step_loss=0.0689]04/11/2024 17:10:29 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-300
Model weights saved in pixart-simpsons-model/checkpoint-300/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:10:29 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-300/optimizer.bin
04/11/2024 17:10:29 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-300/scheduler.bin
04/11/2024 17:10:29 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-300/sampler.bin
04/11/2024 17:10:29 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-300/scaler.pt
04/11/2024 17:10:29 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-300/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-300/pytorch_lora_weights.safetensors
04/11/2024 17:10:29 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-300
Steps:   8%|███████▉                                                                                      | 400/4750 [21:55<3:59:45,  3.31s/it, lr=1e-5, step_loss=0.256]04/11/2024 17:15:55 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-400
Model weights saved in pixart-simpsons-model/checkpoint-400/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:15:55 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-400/optimizer.bin
04/11/2024 17:15:55 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-400/scheduler.bin
04/11/2024 17:15:55 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-400/sampler.bin
04/11/2024 17:15:55 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-400/scaler.pt
04/11/2024 17:15:55 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-400/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-400/pytorch_lora_weights.safetensors
04/11/2024 17:15:55 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-400
Steps:  11%|█████████▊                                                                                   | 500/4750 [27:22<3:54:00,  3.30s/it, lr=1e-5, step_loss=0.0707]04/11/2024 17:21:21 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-500
Model weights saved in pixart-simpsons-model/checkpoint-500/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:21:22 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-500/optimizer.bin
04/11/2024 17:21:22 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-500/scheduler.bin
04/11/2024 17:21:22 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-500/sampler.bin
04/11/2024 17:21:22 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-500/scaler.pt
04/11/2024 17:21:22 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-500/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-500/pytorch_lora_weights.safetensors
04/11/2024 17:21:22 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-500
Steps:  13%|███████████▋                                                                                 | 600/4750 [32:52<3:41:33,  3.20s/it, lr=1e-5, step_loss=0.0838]04/11/2024 17:26:51 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-600
Model weights saved in pixart-simpsons-model/checkpoint-600/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:26:52 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-600/optimizer.bin
04/11/2024 17:26:52 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-600/scheduler.bin
04/11/2024 17:26:52 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-600/sampler.bin
04/11/2024 17:26:52 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-600/scaler.pt
04/11/2024 17:26:52 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-600/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-600/pytorch_lora_weights.safetensors
04/11/2024 17:26:52 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-600
Steps:  15%|█████████████▋                                                                               | 700/4750 [38:19<3:42:00,  3.29s/it, lr=1e-5, step_loss=0.0155]04/11/2024 17:32:18 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-700
Model weights saved in pixart-simpsons-model/checkpoint-700/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:32:18 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-700/optimizer.bin
04/11/2024 17:32:19 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-700/scheduler.bin
04/11/2024 17:32:19 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-700/sampler.bin
04/11/2024 17:32:19 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-700/scaler.pt
04/11/2024 17:32:19 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-700/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-700/pytorch_lora_weights.safetensors
04/11/2024 17:32:19 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-700
Steps:  17%|███████████████▍                                                                            | 800/4750 [43:49<3:36:07,  3.28s/it, lr=1e-5, step_loss=0.00618]04/11/2024 17:37:48 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-800
Model weights saved in pixart-simpsons-model/checkpoint-800/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:37:48 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-800/optimizer.bin
04/11/2024 17:37:48 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-800/scheduler.bin
04/11/2024 17:37:48 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-800/sampler.bin
04/11/2024 17:37:48 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-800/scaler.pt
04/11/2024 17:37:48 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-800/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-800/pytorch_lora_weights.safetensors
04/11/2024 17:37:49 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-800
Steps:  19%|█████████████████▊                                                                            | 900/4750 [49:17<3:37:34,  3.39s/it, lr=1e-5, step_loss=0.242]04/11/2024 17:43:17 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-900
Model weights saved in pixart-simpsons-model/checkpoint-900/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:43:17 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-900/optimizer.bin
04/11/2024 17:43:17 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-900/scheduler.bin
04/11/2024 17:43:17 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-900/sampler.bin
04/11/2024 17:43:17 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-900/scaler.pt
04/11/2024 17:43:17 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-900/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-900/pytorch_lora_weights.safetensors
04/11/2024 17:43:17 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-900
Steps:  21%|███████████████████▊                                                                          | 1000/4750 [54:47<3:26:41,  3.31s/it, lr=1e-5, step_loss=0.11]04/11/2024 17:48:47 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1000
Model weights saved in pixart-simpsons-model/checkpoint-1000/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:48:47 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1000/optimizer.bin
04/11/2024 17:48:47 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1000/scheduler.bin
04/11/2024 17:48:47 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1000/sampler.bin
04/11/2024 17:48:47 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1000/scaler.pt
04/11/2024 17:48:47 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1000/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1000/pytorch_lora_weights.safetensors
04/11/2024 17:48:47 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1000
Steps:  23%|████████████████████▊                                                                     | 1100/4750 [1:00:18<3:22:12,  3.32s/it, lr=1e-5, step_loss=0.0415]04/11/2024 17:54:17 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 17:54:17 - INFO - __main__ - removing checkpoints: checkpoint-100
04/11/2024 17:54:17 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1100
Model weights saved in pixart-simpsons-model/checkpoint-1100/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:54:18 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1100/optimizer.bin
04/11/2024 17:54:18 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1100/scheduler.bin
04/11/2024 17:54:18 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1100/sampler.bin
04/11/2024 17:54:18 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1100/scaler.pt
04/11/2024 17:54:18 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1100/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1100/pytorch_lora_weights.safetensors
04/11/2024 17:54:18 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1100
Steps:  25%|██████████████████████▉                                                                    | 1200/4750 [1:05:51<3:16:01,  3.31s/it, lr=1e-5, step_loss=0.028]04/11/2024 17:59:50 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 17:59:50 - INFO - __main__ - removing checkpoints: checkpoint-200
04/11/2024 17:59:50 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1200
Model weights saved in pixart-simpsons-model/checkpoint-1200/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 17:59:51 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1200/optimizer.bin
04/11/2024 17:59:51 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1200/scheduler.bin
04/11/2024 17:59:51 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1200/sampler.bin
04/11/2024 17:59:51 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1200/scaler.pt
04/11/2024 17:59:51 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1200/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1200/pytorch_lora_weights.safetensors
04/11/2024 17:59:51 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1200
Steps:  27%|████████████████████████▋                                                                 | 1300/4750 [1:11:23<3:16:49,  3.42s/it, lr=1e-5, step_loss=0.0215]04/11/2024 18:05:22 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:05:22 - INFO - __main__ - removing checkpoints: checkpoint-300
04/11/2024 18:05:22 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1300
Model weights saved in pixart-simpsons-model/checkpoint-1300/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:05:22 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1300/optimizer.bin
04/11/2024 18:05:22 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1300/scheduler.bin
04/11/2024 18:05:22 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1300/sampler.bin
04/11/2024 18:05:23 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1300/scaler.pt
04/11/2024 18:05:23 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1300/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1300/pytorch_lora_weights.safetensors
04/11/2024 18:05:23 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1300
Steps:  29%|██████████████████████████▌                                                               | 1400/4750 [1:16:56<3:09:09,  3.39s/it, lr=1e-5, step_loss=0.0645]04/11/2024 18:10:55 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:10:55 - INFO - __main__ - removing checkpoints: checkpoint-400
04/11/2024 18:10:55 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1400
Model weights saved in pixart-simpsons-model/checkpoint-1400/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:10:56 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1400/optimizer.bin
04/11/2024 18:10:56 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1400/scheduler.bin
04/11/2024 18:10:56 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1400/sampler.bin
04/11/2024 18:10:56 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1400/scaler.pt
04/11/2024 18:10:56 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1400/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1400/pytorch_lora_weights.safetensors
04/11/2024 18:10:56 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1400
Steps:  32%|████████████████████████████▋                                                              | 1500/4750 [1:22:30<3:04:58,  3.41s/it, lr=1e-5, step_loss=0.102]04/11/2024 18:16:30 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:16:30 - INFO - __main__ - removing checkpoints: checkpoint-500
04/11/2024 18:16:30 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1500
Model weights saved in pixart-simpsons-model/checkpoint-1500/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:16:30 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1500/optimizer.bin
04/11/2024 18:16:30 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1500/scheduler.bin
04/11/2024 18:16:30 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1500/sampler.bin
04/11/2024 18:16:30 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1500/scaler.pt
04/11/2024 18:16:30 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1500/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1500/pytorch_lora_weights.safetensors
04/11/2024 18:16:30 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1500
Steps:  34%|██████████████████████████████▎                                                           | 1600/4750 [1:28:04<3:01:33,  3.46s/it, lr=1e-5, step_loss=0.0559]04/11/2024 18:22:03 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:22:03 - INFO - __main__ - removing checkpoints: checkpoint-600
04/11/2024 18:22:03 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1600
Model weights saved in pixart-simpsons-model/checkpoint-1600/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:22:03 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1600/optimizer.bin
04/11/2024 18:22:03 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1600/scheduler.bin
04/11/2024 18:22:04 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1600/sampler.bin
04/11/2024 18:22:04 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1600/scaler.pt
04/11/2024 18:22:04 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1600/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1600/pytorch_lora_weights.safetensors
04/11/2024 18:22:04 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1600
Steps:  36%|████████████████████████████████▏                                                         | 1700/4750 [1:33:42<2:52:48,  3.40s/it, lr=1e-5, step_loss=0.0607]04/11/2024 18:27:41 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:27:41 - INFO - __main__ - removing checkpoints: checkpoint-700
04/11/2024 18:27:42 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1700
Model weights saved in pixart-simpsons-model/checkpoint-1700/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:27:42 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1700/optimizer.bin
04/11/2024 18:27:42 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1700/scheduler.bin
04/11/2024 18:27:42 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1700/sampler.bin
04/11/2024 18:27:42 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1700/scaler.pt
04/11/2024 18:27:42 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1700/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1700/pytorch_lora_weights.safetensors
04/11/2024 18:27:42 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1700
Steps:  38%|██████████████████████████████████                                                        | 1800/4750 [1:39:20<2:46:11,  3.38s/it, lr=1e-5, step_loss=0.0139]04/11/2024 18:33:19 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:33:19 - INFO - __main__ - removing checkpoints: checkpoint-800
04/11/2024 18:33:19 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1800
Model weights saved in pixart-simpsons-model/checkpoint-1800/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:33:19 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1800/optimizer.bin
04/11/2024 18:33:19 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1800/scheduler.bin
04/11/2024 18:33:19 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1800/sampler.bin
04/11/2024 18:33:20 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1800/scaler.pt
04/11/2024 18:33:20 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1800/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1800/pytorch_lora_weights.safetensors
04/11/2024 18:33:20 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1800
Steps:  40%|████████████████████████████████████▊                                                       | 1900/4750 [1:44:57<2:09:58,  2.74s/it, lr=1e-5, step_loss=0.32]04/11/2024 18:38:56 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:38:57 - INFO - __main__ - removing checkpoints: checkpoint-900
04/11/2024 18:38:57 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-1900
Model weights saved in pixart-simpsons-model/checkpoint-1900/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:38:57 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-1900/optimizer.bin
04/11/2024 18:38:57 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-1900/scheduler.bin
04/11/2024 18:38:57 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-1900/sampler.bin
04/11/2024 18:38:57 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-1900/scaler.pt
04/11/2024 18:38:57 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-1900/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-1900/pytorch_lora_weights.safetensors
04/11/2024 18:38:57 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-1900
Steps:  42%|█████████████████████████████████████▉                                                    | 2000/4750 [1:50:37<2:32:05,  3.32s/it, lr=1e-5, step_loss=0.0289]04/11/2024 18:44:36 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:44:36 - INFO - __main__ - removing checkpoints: checkpoint-1000
04/11/2024 18:44:36 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2000
Model weights saved in pixart-simpsons-model/checkpoint-2000/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:44:37 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2000/optimizer.bin
04/11/2024 18:44:37 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2000/scheduler.bin
04/11/2024 18:44:37 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2000/sampler.bin
04/11/2024 18:44:37 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2000/scaler.pt
04/11/2024 18:44:37 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2000/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2000/pytorch_lora_weights.safetensors
04/11/2024 18:44:37 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2000
Steps:  44%|███████████████████████████████████████▎                                                 | 2100/4750 [1:56:19<2:30:28,  3.41s/it, lr=1e-5, step_loss=0.00492]04/11/2024 18:50:18 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:50:18 - INFO - __main__ - removing checkpoints: checkpoint-1100
04/11/2024 18:50:18 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2100
Model weights saved in pixart-simpsons-model/checkpoint-2100/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:50:18 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2100/optimizer.bin
04/11/2024 18:50:18 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2100/scheduler.bin
04/11/2024 18:50:18 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2100/sampler.bin
04/11/2024 18:50:18 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2100/scaler.pt
04/11/2024 18:50:18 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2100/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2100/pytorch_lora_weights.safetensors
04/11/2024 18:50:19 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2100
Steps:  46%|█████████████████████████████████████████▋                                                | 2200/4750 [2:02:03<2:23:54,  3.39s/it, lr=1e-5, step_loss=0.0491]04/11/2024 18:56:02 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 18:56:02 - INFO - __main__ - removing checkpoints: checkpoint-1200
04/11/2024 18:56:02 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2200
Model weights saved in pixart-simpsons-model/checkpoint-2200/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 18:56:02 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2200/optimizer.bin
04/11/2024 18:56:02 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2200/scheduler.bin
04/11/2024 18:56:02 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2200/sampler.bin
04/11/2024 18:56:03 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2200/scaler.pt
04/11/2024 18:56:03 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2200/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2200/pytorch_lora_weights.safetensors
04/11/2024 18:56:03 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2200
Steps:  48%|████████████████████████████████████████████                                               | 2300/4750 [2:07:48<2:26:28,  3.59s/it, lr=1e-5, step_loss=0.159]04/11/2024 19:01:47 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:01:47 - INFO - __main__ - removing checkpoints: checkpoint-1300
04/11/2024 19:01:47 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2300
Model weights saved in pixart-simpsons-model/checkpoint-2300/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:01:48 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2300/optimizer.bin
04/11/2024 19:01:48 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2300/scheduler.bin
04/11/2024 19:01:48 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2300/sampler.bin
04/11/2024 19:01:48 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2300/scaler.pt
04/11/2024 19:01:48 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2300/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2300/pytorch_lora_weights.safetensors
04/11/2024 19:01:48 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2300
Steps:  51%|█████████████████████████████████████████████▍                                            | 2400/4750 [2:13:34<2:16:58,  3.50s/it, lr=1e-5, step_loss=0.0367]04/11/2024 19:07:34 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:07:34 - INFO - __main__ - removing checkpoints: checkpoint-1400
04/11/2024 19:07:34 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2400
Model weights saved in pixart-simpsons-model/checkpoint-2400/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:07:34 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2400/optimizer.bin
04/11/2024 19:07:34 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2400/scheduler.bin
04/11/2024 19:07:34 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2400/sampler.bin
04/11/2024 19:07:34 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2400/scaler.pt
04/11/2024 19:07:34 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2400/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2400/pytorch_lora_weights.safetensors
04/11/2024 19:07:34 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2400
Steps:  53%|███████████████████████████████████████████████▎                                          | 2500/4750 [2:19:22<2:15:46,  3.62s/it, lr=1e-5, step_loss=0.0786]04/11/2024 19:13:21 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:13:21 - INFO - __main__ - removing checkpoints: checkpoint-1500
04/11/2024 19:13:21 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2500
Model weights saved in pixart-simpsons-model/checkpoint-2500/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:13:22 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2500/optimizer.bin
04/11/2024 19:13:22 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2500/scheduler.bin
04/11/2024 19:13:22 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2500/sampler.bin
04/11/2024 19:13:22 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2500/scaler.pt
04/11/2024 19:13:22 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2500/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2500/pytorch_lora_weights.safetensors
04/11/2024 19:13:22 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2500
Steps:  55%|█████████████████████████████████████████████████▎                                        | 2600/4750 [2:25:12<2:03:13,  3.44s/it, lr=1e-5, step_loss=0.0613]04/11/2024 19:19:12 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:19:12 - INFO - __main__ - removing checkpoints: checkpoint-1600
04/11/2024 19:19:12 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2600
Model weights saved in pixart-simpsons-model/checkpoint-2600/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:19:12 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2600/optimizer.bin
04/11/2024 19:19:12 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2600/scheduler.bin
04/11/2024 19:19:12 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2600/sampler.bin
04/11/2024 19:19:12 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2600/scaler.pt
04/11/2024 19:19:12 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2600/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2600/pytorch_lora_weights.safetensors
04/11/2024 19:19:12 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2600
Steps:  57%|███████████████████████████████████████████████████▏                                      | 2700/4750 [2:31:02<1:58:56,  3.48s/it, lr=1e-5, step_loss=0.0143]04/11/2024 19:25:01 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:25:01 - INFO - __main__ - removing checkpoints: checkpoint-1700
04/11/2024 19:25:01 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2700
Model weights saved in pixart-simpsons-model/checkpoint-2700/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:25:02 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2700/optimizer.bin
04/11/2024 19:25:02 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2700/scheduler.bin
04/11/2024 19:25:02 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2700/sampler.bin
04/11/2024 19:25:02 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2700/scaler.pt
04/11/2024 19:25:02 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2700/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2700/pytorch_lora_weights.safetensors
04/11/2024 19:25:02 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2700
Steps:  59%|█████████████████████████████████████████████████████                                     | 2800/4750 [2:36:54<1:51:59,  3.45s/it, lr=1e-5, step_loss=0.0103]04/11/2024 19:30:53 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:30:53 - INFO - __main__ - removing checkpoints: checkpoint-1800
04/11/2024 19:30:53 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2800
Model weights saved in pixart-simpsons-model/checkpoint-2800/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:30:54 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2800/optimizer.bin
04/11/2024 19:30:54 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2800/scheduler.bin
04/11/2024 19:30:54 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2800/sampler.bin
04/11/2024 19:30:54 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2800/scaler.pt
04/11/2024 19:30:54 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2800/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2800/pytorch_lora_weights.safetensors
04/11/2024 19:30:54 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2800
Steps:  61%|█████████████████████████████████████████████████████▋                                  | 2900/4750 [2:43:42<1:50:05,  3.57s/it, lr=1e-5, step_loss=0.000983]04/11/2024 19:37:41 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:37:42 - INFO - __main__ - removing checkpoints: checkpoint-1900
04/11/2024 19:37:42 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-2900
Model weights saved in pixart-simpsons-model/checkpoint-2900/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:37:42 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-2900/optimizer.bin
04/11/2024 19:37:42 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-2900/scheduler.bin
04/11/2024 19:37:42 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-2900/sampler.bin
04/11/2024 19:37:42 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-2900/scaler.pt
04/11/2024 19:37:42 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-2900/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-2900/pytorch_lora_weights.safetensors
04/11/2024 19:37:42 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-2900
Steps:  63%|████████████████████████████████████████████████████████▊                                 | 3000/4750 [2:49:38<1:44:29,  3.58s/it, lr=1e-5, step_loss=0.0698]04/11/2024 19:43:37 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:43:37 - INFO - __main__ - removing checkpoints: checkpoint-2000
04/11/2024 19:43:37 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3000
Model weights saved in pixart-simpsons-model/checkpoint-3000/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:43:37 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3000/optimizer.bin
04/11/2024 19:43:37 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3000/scheduler.bin
04/11/2024 19:43:37 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3000/sampler.bin
04/11/2024 19:43:37 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3000/scaler.pt
04/11/2024 19:43:37 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3000/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3000/pytorch_lora_weights.safetensors
04/11/2024 19:43:38 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3000
Steps:  65%|██████████████████████████████████████████████████████████▋                               | 3100/4750 [2:55:34<1:40:30,  3.65s/it, lr=1e-5, step_loss=0.0878]04/11/2024 19:49:33 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:49:33 - INFO - __main__ - removing checkpoints: checkpoint-2100
04/11/2024 19:49:33 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3100
Model weights saved in pixart-simpsons-model/checkpoint-3100/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:49:33 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3100/optimizer.bin
04/11/2024 19:49:34 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3100/scheduler.bin
04/11/2024 19:49:34 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3100/sampler.bin
04/11/2024 19:49:34 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3100/scaler.pt
04/11/2024 19:49:34 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3100/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3100/pytorch_lora_weights.safetensors
04/11/2024 19:49:34 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3100
Steps:  67%|████████████████████████████████████████████████████████████▋                             | 3200/4750 [3:01:32<1:35:31,  3.70s/it, lr=1e-5, step_loss=0.0155]04/11/2024 19:55:31 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 19:55:32 - INFO - __main__ - removing checkpoints: checkpoint-2200
04/11/2024 19:55:32 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3200
Model weights saved in pixart-simpsons-model/checkpoint-3200/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 19:55:32 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3200/optimizer.bin
04/11/2024 19:55:32 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3200/scheduler.bin
04/11/2024 19:55:32 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3200/sampler.bin
04/11/2024 19:55:32 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3200/scaler.pt
04/11/2024 19:55:32 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3200/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3200/pytorch_lora_weights.safetensors
04/11/2024 19:55:32 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3200
Steps:  69%|██████████████████████████████████████████████████████████████▌                           | 3300/4750 [3:07:33<1:25:38,  3.54s/it, lr=1e-5, step_loss=0.0723]04/11/2024 20:01:32 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:01:32 - INFO - __main__ - removing checkpoints: checkpoint-2300
04/11/2024 20:01:32 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3300
Model weights saved in pixart-simpsons-model/checkpoint-3300/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:01:33 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3300/optimizer.bin
04/11/2024 20:01:33 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3300/scheduler.bin
04/11/2024 20:01:33 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3300/sampler.bin
04/11/2024 20:01:33 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3300/scaler.pt
04/11/2024 20:01:33 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3300/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3300/pytorch_lora_weights.safetensors
04/11/2024 20:01:33 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3300
Steps:  72%|███████████████████████████████████████████████████████████████▋                         | 3400/4750 [3:13:35<1:25:02,  3.78s/it, lr=1e-5, step_loss=0.00143]04/11/2024 20:07:34 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:07:34 - INFO - __main__ - removing checkpoints: checkpoint-2400
04/11/2024 20:07:34 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3400
Model weights saved in pixart-simpsons-model/checkpoint-3400/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:07:34 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3400/optimizer.bin
04/11/2024 20:07:35 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3400/scheduler.bin
04/11/2024 20:07:35 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3400/sampler.bin
04/11/2024 20:07:35 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3400/scaler.pt
04/11/2024 20:07:35 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3400/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3400/pytorch_lora_weights.safetensors
04/11/2024 20:07:35 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3400
Steps:  74%|█████████████████████████████████████████████████████████████████▌                       | 3500/4750 [3:19:36<1:15:29,  3.62s/it, lr=1e-5, step_loss=0.00604]04/11/2024 20:13:36 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:13:36 - INFO - __main__ - removing checkpoints: checkpoint-2500
04/11/2024 20:13:36 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3500
Model weights saved in pixart-simpsons-model/checkpoint-3500/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:13:36 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3500/optimizer.bin
04/11/2024 20:13:36 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3500/scheduler.bin
04/11/2024 20:13:36 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3500/sampler.bin
04/11/2024 20:13:36 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3500/scaler.pt
04/11/2024 20:13:36 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3500/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3500/pytorch_lora_weights.safetensors
04/11/2024 20:13:36 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3500
Steps:  76%|████████████████████████████████████████████████████████████████████▏                     | 3600/4750 [3:25:37<1:10:08,  3.66s/it, lr=1e-5, step_loss=0.0555]04/11/2024 20:19:37 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:19:37 - INFO - __main__ - removing checkpoints: checkpoint-2600
04/11/2024 20:19:37 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3600
Model weights saved in pixart-simpsons-model/checkpoint-3600/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:19:37 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3600/optimizer.bin
04/11/2024 20:19:37 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3600/scheduler.bin
04/11/2024 20:19:37 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3600/sampler.bin
04/11/2024 20:19:37 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3600/scaler.pt
04/11/2024 20:19:37 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3600/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3600/pytorch_lora_weights.safetensors
04/11/2024 20:19:37 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3600
Steps:  78%|██████████████████████████████████████████████████████████████████████                    | 3700/4750 [3:31:39<1:04:29,  3.69s/it, lr=1e-5, step_loss=0.0312]04/11/2024 20:25:38 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:25:39 - INFO - __main__ - removing checkpoints: checkpoint-2700
04/11/2024 20:25:39 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3700
Model weights saved in pixart-simpsons-model/checkpoint-3700/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:25:39 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3700/optimizer.bin
04/11/2024 20:25:39 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3700/scheduler.bin
04/11/2024 20:25:39 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3700/sampler.bin
04/11/2024 20:25:39 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3700/scaler.pt
04/11/2024 20:25:39 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3700/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3700/pytorch_lora_weights.safetensors
04/11/2024 20:25:39 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3700
Steps:  80%|████████████████████████████████████████████████████████████████████████▊                  | 3800/4750 [3:37:42<47:55,  3.03s/it, lr=1e-5, step_loss=0.00381]04/11/2024 20:31:42 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:31:42 - INFO - __main__ - removing checkpoints: checkpoint-2800
04/11/2024 20:31:42 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3800
Model weights saved in pixart-simpsons-model/checkpoint-3800/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:31:42 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3800/optimizer.bin
04/11/2024 20:31:42 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3800/scheduler.bin
04/11/2024 20:31:42 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3800/sampler.bin
04/11/2024 20:31:42 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3800/scaler.pt
04/11/2024 20:31:42 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3800/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3800/pytorch_lora_weights.safetensors
04/11/2024 20:31:42 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3800
Steps:  82%|████████████████████████████████████████████████████████████████████████████▎                | 3900/4750 [3:43:47<51:15,  3.62s/it, lr=1e-5, step_loss=0.368]04/11/2024 20:37:46 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:37:46 - INFO - __main__ - removing checkpoints: checkpoint-2900
04/11/2024 20:37:46 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-3900
Model weights saved in pixart-simpsons-model/checkpoint-3900/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:37:46 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-3900/optimizer.bin
04/11/2024 20:37:46 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-3900/scheduler.bin
04/11/2024 20:37:46 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-3900/sampler.bin
04/11/2024 20:37:47 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-3900/scaler.pt
04/11/2024 20:37:47 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-3900/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-3900/pytorch_lora_weights.safetensors
04/11/2024 20:37:47 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-3900
Steps:  84%|██████████████████████████████████████████████████████████████████████████████▎              | 4000/4750 [3:49:55<45:56,  3.68s/it, lr=1e-5, step_loss=0.146]04/11/2024 20:43:54 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:43:54 - INFO - __main__ - removing checkpoints: checkpoint-3000
04/11/2024 20:43:55 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4000
Model weights saved in pixart-simpsons-model/checkpoint-4000/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:43:55 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4000/optimizer.bin
04/11/2024 20:43:55 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4000/scheduler.bin
04/11/2024 20:43:55 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4000/sampler.bin
04/11/2024 20:43:55 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4000/scaler.pt
04/11/2024 20:43:55 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4000/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4000/pytorch_lora_weights.safetensors
04/11/2024 20:43:55 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4000
Steps:  86%|███████████████████████████████████████████████████████████████████████████████▍            | 4100/4750 [3:56:03<39:27,  3.64s/it, lr=1e-5, step_loss=0.0432]04/11/2024 20:50:02 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:50:02 - INFO - __main__ - removing checkpoints: checkpoint-3100
04/11/2024 20:50:02 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4100
Model weights saved in pixart-simpsons-model/checkpoint-4100/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:50:02 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4100/optimizer.bin
04/11/2024 20:50:03 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4100/scheduler.bin
04/11/2024 20:50:03 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4100/sampler.bin
04/11/2024 20:50:03 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4100/scaler.pt
04/11/2024 20:50:03 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4100/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4100/pytorch_lora_weights.safetensors
04/11/2024 20:50:03 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4100
Steps:  88%|█████████████████████████████████████████████████████████████████████████████████▎          | 4200/4750 [4:02:12<33:13,  3.62s/it, lr=1e-5, step_loss=0.0885]04/11/2024 20:56:12 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 20:56:12 - INFO - __main__ - removing checkpoints: checkpoint-3200
04/11/2024 20:56:12 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4200
Model weights saved in pixart-simpsons-model/checkpoint-4200/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 20:56:12 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4200/optimizer.bin
04/11/2024 20:56:12 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4200/scheduler.bin
04/11/2024 20:56:12 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4200/sampler.bin
04/11/2024 20:56:12 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4200/scaler.pt
04/11/2024 20:56:12 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4200/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4200/pytorch_lora_weights.safetensors
04/11/2024 20:56:12 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4200
Steps:  91%|████████████████████████████████████████████████████████████████████████████████████▏        | 4300/4750 [4:10:15<52:26,  6.99s/it, lr=1e-5, step_loss=0.393]04/11/2024 21:04:14 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 21:04:14 - INFO - __main__ - removing checkpoints: checkpoint-3300
04/11/2024 21:04:14 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4300
Model weights saved in pixart-simpsons-model/checkpoint-4300/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 21:04:15 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4300/optimizer.bin
04/11/2024 21:04:15 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4300/scheduler.bin
04/11/2024 21:04:15 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4300/sampler.bin
04/11/2024 21:04:15 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4300/scaler.pt
04/11/2024 21:04:15 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4300/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4300/pytorch_lora_weights.safetensors
04/11/2024 21:04:15 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4300
Steps:  93%|██████████████████████████████████████████████████████████████████████████████████████▏      | 4400/4750 [4:19:40<33:43,  5.78s/it, lr=1e-5, step_loss=0.107]04/11/2024 21:13:40 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 21:13:40 - INFO - __main__ - removing checkpoints: checkpoint-3400
04/11/2024 21:13:40 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4400
Model weights saved in pixart-simpsons-model/checkpoint-4400/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 21:13:40 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4400/optimizer.bin
04/11/2024 21:13:41 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4400/scheduler.bin
04/11/2024 21:13:41 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4400/sampler.bin
04/11/2024 21:13:41 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4400/scaler.pt
04/11/2024 21:13:41 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4400/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4400/pytorch_lora_weights.safetensors
04/11/2024 21:13:41 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4400
Steps:  95%|█████████████████████████████████████████████████████████████████████████████████████████     | 4500/4750 [4:29:13<23:55,  5.74s/it, lr=1e-5, step_loss=0.12]04/11/2024 21:23:12 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 21:23:12 - INFO - __main__ - removing checkpoints: checkpoint-3500
04/11/2024 21:23:12 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4500
Model weights saved in pixart-simpsons-model/checkpoint-4500/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 21:23:13 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4500/optimizer.bin
04/11/2024 21:23:13 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4500/scheduler.bin
04/11/2024 21:23:13 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4500/sampler.bin
04/11/2024 21:23:13 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4500/scaler.pt
04/11/2024 21:23:13 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4500/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4500/pytorch_lora_weights.safetensors
04/11/2024 21:23:13 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4500
Steps:  97%|█████████████████████████████████████████████████████████████████████████████████████████   | 4600/4750 [4:38:44<14:06,  5.64s/it, lr=1e-5, step_loss=0.0277]04/11/2024 21:32:43 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 21:32:43 - INFO - __main__ - removing checkpoints: checkpoint-3600
04/11/2024 21:32:43 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4600
Model weights saved in pixart-simpsons-model/checkpoint-4600/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 21:32:44 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4600/optimizer.bin
04/11/2024 21:32:44 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4600/scheduler.bin
04/11/2024 21:32:44 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4600/sampler.bin
04/11/2024 21:32:44 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4600/scaler.pt
04/11/2024 21:32:44 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4600/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4600/pytorch_lora_weights.safetensors
04/11/2024 21:32:44 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4600
Steps:  99%|████████████████████████████████████████████████████████████████████████████████████████████ | 4700/4750 [4:48:34<09:17, 11.15s/it, lr=1e-5, step_loss=0.047]04/11/2024 21:42:34 - INFO - __main__ - 10 checkpoints already exist, removing 1 checkpoints
04/11/2024 21:42:35 - INFO - __main__ - removing checkpoints: checkpoint-3700
04/11/2024 21:42:35 - INFO - accelerate.accelerator - Saving current state to pixart-simpsons-model/checkpoint-4700
Model weights saved in pixart-simpsons-model/checkpoint-4700/transformer_lora/pytorch_lora_weights.safetensors
04/11/2024 21:42:38 - INFO - accelerate.checkpointing - Optimizer state saved in pixart-simpsons-model/checkpoint-4700/optimizer.bin
04/11/2024 21:42:39 - INFO - accelerate.checkpointing - Scheduler state saved in pixart-simpsons-model/checkpoint-4700/scheduler.bin
04/11/2024 21:42:39 - INFO - accelerate.checkpointing - Sampler state for dataloader 0 saved in pixart-simpsons-model/checkpoint-4700/sampler.bin
04/11/2024 21:42:39 - INFO - accelerate.checkpointing - Gradient scaler state saved in pixart-simpsons-model/checkpoint-4700/scaler.pt
04/11/2024 21:42:40 - INFO - accelerate.checkpointing - Random states saved in pixart-simpsons-model/checkpoint-4700/random_states_0.pkl
Model weights saved in pixart-simpsons-model/checkpoint-4700/pytorch_lora_weights.safetensors
04/11/2024 21:42:41 - INFO - __main__ - Saved state to pixart-simpsons-model/checkpoint-4700
Steps: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 4750/4750 [4:53:43<00:00,  4.71s/it, lr=1e-5, step_loss=0.0997]Model weights saved in pixart-simpsons-model/transformer_lora/pytorch_lora_weights.safetensors
Steps: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 4750/4750 [4:53:43<00:00,  3.71s/it, lr=1e-5, step_loss=0.0997]

[lawrence-cj]

Any log about the loss. That would be more helpful.

[raulc0399]

the loss is in the logs above at the end of the steps progress bar
4750/4750 [4:53:43<00:00, 3.71s/it, lr=1e-5, step_loss=0.0997

unfortunately the tensorboard file seems to be corrupt :(

[raulc0399]

Hi @lawrence-cj

here a log of a new training i am running - it does not change event if i try with other LR values:

[raulc0399]

it started working once i applyed the patch that i mention here
https://github.com/orgs/PixArt-alpha/discussions/119#discussioncomment-9129986